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ABSTRACT 


Modeling and Simulation (M&S) technology uses models to develop data as a 
basis for making managerial or technical decisions. M&S can be a valuable tool for 
decision-makers but it is usually under-used. The United States Army Developmental 
Test Command (DTC) is leveraging M&S to accomplish its missions through the Virtual 
Proving Ground (VPG) Program. DTC supplies a customer decision-maker, usually a 
Program Manager (PM), with data on the cost-effectiveness of new virtual and physical 
test technologies in order to plan test activities. DTC requires a methodology to develop 
a business plan that supports the use of M&S and to provide a cost benefit analysis of 
particular virtual test capabilities. DTC commissioned independent studies of past test 
programs to estimate the costs to achieve the same scope of testing, as tested using 
available virtual test techniques and as using previous, less VPG-intensive test 
methodologies. The studies showed that virtual testing provided significant cost benefits 
to each PM. An objective is to examine cost avoidance results from those studies and 
additional data with a methodology consistent with current cost estimation guidance to 
determine a return on investment relationship. This thesis will endeavor to establish an 
equitable methodology for accounting or realizing the direct benefits associated with 
using M&S in testing. The details of the steps will be developed as necessary to perform 
a business case analysis. 
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L INTRODUCTION 


A. PURPOSE 

The research area of this thesis will be Modeling and Simulation (M&S) Policy as 
it pertains to the economics of its application in the acquisition discipline of 
developmental testing. 

The US Army Developmental Test Command (DTC) has the missions of 
developmental technical testing and safety certification of all Army materiel. A test 
might be a live test of physical hardware in a physical environment or a virtual test of 
simulated, and/or physical hardware in a simulated environment. DTC is leveraging 
M&S to accomplish its missions through the Virtual Proving Ground (VPG) Program. 
DTC supplies a customer decision-maker, usually a Program Manager (PM), with data on 
the cost-effectiveness of new virtual and physical test technologies. DTC requires a 
methodology to develop a business plan that supports the use of M&S and to capture the 
cost avoidance of particular virtual tests consistent with current Army cost estimation 
procedures. DTC commissioned independent studies of past test programs to estimate 
the costs to achieve the same scope of testing, as tested using available virtual test 
techniques and as using previous, less VPG-intensive test methodologies. The studies 
showed that virtual testing provided significant cost benefits to each PM. An objective is 
to examine cost avoidance results from those studies and additional data with a 
methodology consistent with current cost estimation guidance to determine a return on 
investment relationship. 
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B. BACKGROUND 


The earliest use of modeling can be traced back to 1867 with the Rigid Kriegspiel 
(War Game). M&S have been used in the design of weapon systems for many years. 
The testing community has been using M&S techniques for the at least the last ten years 
to enable testing that was previously not possible or economical. The testing community 
needs a consistent technique to quantify the benefit of using M&S. This benefit can then 
help prove that sound decisions are being made for using M&S. More particularly, the 
developmental test community needs a technique since they are integral part of assessing 
whether the performance of a weapon system is mature and ready for production. The 
developmental testing community is constantly working with the research and 
development community that is using M&S extensively. With the advent of the use of 
M&S as directed by Department of Defense (DoD) policy, the costs and associated 
benefits with the development and use of M&S techniques fit traditional cost benefit 
models with modification and much emphasis on indirect benefits. Commonly, the 
decision to employ M&S comes as a result of policy direction rather than economic 
benefit. There is no documented or rigorous technique to account for the benefit realized 
when using M&S in the developmental testing process. This thesis will endeavor to 
establish an equitable methodology for accounting or realizing the direct benefits 
associated with using M&S in testing. The details of the steps will be developed as 
necessary to perform a business case analysis. 

While there are some common applications industry-wide, DoD is not focused on 
profit, but on performance and total ownership costs. The 1993 Government 
Performance and Results Act (GPRA) has put pressure on PMs by putting in place a 
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results-based management regime intended to tie funding decision directly to program 
performance (Laurent, 2000). The GPRA requires organizations to set outcome goals, 
measure their performance and report their accomplishments. These reports to be 
developed by PMs must be coupled with cost data and based on credible information to 
enable decision-makers to make funding decisions. The GPRA is going to push agencies 
to clarify their business lines and to collect performance and cost information. With this 
information, PMs will be better able to build “better business cases for maintaining 
current programs or moving to more effective methods for achieving results” (Laurent, 
2000). A recent Industrial College of the Armed Forces (ICAF) team surveyed a large 
number of DoD PMs with programs in various stages of the acquisition process. The 
survey requested general information from the PMs regarding how they made decisions 
on M&S investments. In an effort to maintain balance, they also surveyed several 
Government contractors. Additionally, they also conducted a literature search to capture 
the body of knowledge in non-Govemment organizations related to justifying M&S 
investments. 

The ICAF team published their efforts in the Fall 2000 issue of the Acquisition 
Review Quarterly. The article provides a business _ case framework for Program 
Managers (PMs) within the Department of Defense to use when determining how to 
apply modeling and simulation in project management (Brown, Grant, Kotchman, 
Reyenga, & Szanto, 2000). This thesis will build upon this framework to develop one for 
developmental testing. A business case analysis is a methodology to evaluate investment 
opportunities. M&S employ models, either statically or over time, to develop data as a 
basis for making managerial or technical decisions. Models are physical, mathematical 
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or logical representations of a system, entity, phenomenon, or process (DoD, 1997). 
Simulations are methods for implementing models over time. Normally, we associate 
simulations with a software program that implements models over time, within the 
context of a given scenario. Simulations permit the user to assess variables and the 
predictability of a single or series of outcomes. 

The use of M&S is widely misunderstood within the DoD. Recent Government 
direction to use Simulation Based Acquisition (SBA) in DoD programs is an example of 
a policy with good intentions but poorly shaped execution. This edict has been met with, 
at best, marginal acceptance, and at worst, abject resentment. Such resentment and 
apprehension spring from institutionalized biases, including DoD funding procedures, 
that work against optimizing the potential gains of employing modeling and simulation. 
By far, the severest criticisms targeted at M&S center on the debate over Return on 
Investment (ROI). ROI is just one of many techniques for evaluating the use of M&S in 
a program. Ostensibly, DoD has accepted this new technology as a means of reducing 
costs, increasing cost avoidance and banking the residual benefits for other projects. 
Seldom is the PM or testing community given sufficient funds, staff, or time to 
investigate the potential benefits of tools or technologies such as M&S. Importantly, 
leadership provides little incentive to capture data, build expensive models or conduct 
additional analyses to transfer M&S results to other projects. Simply put, everyone is 
under intense pressure to complete his or her programs on or under budget and within 
time lines. Existing programs lack enticement to develop new models or simulation tools 
that may have wider application to other programs, or that will be much cheaper to 
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operate and sustain. With few exceptions, these occurrences were more a result of 
coincidence than deliberateness. 

Perhaps the greatest impediment to M&S acceptance is lack of knowledge. Many 
people do not understand the potential benefits of this technology, or how to define needs 
and produce the right tools that will help the project. Unfortunately, this reticence may 
be reinforced by an institution which neither favors nor rewards risk takers. Sometimes, 
a PM may not know for certain if a major investment in M&S is warranted. If he wants 
to invest in the technology “just to see if there is a benefit” he faces criticism if the results 
are not positive. Thus, he must weigh the costs of the risk, and consequently few take the 
chance, preferring more traditional approaches like the building of expensive mock ups; 
the use of labor, time and money intensive trials, and incurred costs of waste. 

While all of this bodes well for an anecdotal argument supporting M&S, what’s 
needed is a more reasoned and defensible process for determining M&S investments. 
While few refute the intuitive benefits of M&S, PMs quite rightly argue that any tool 
must be first measured against its potential benefits before it is used. Unfortunately, the 
question is not easily answered. It depends on a variety of factors, including the project’s 
funding, time period to recoup the investment, and perceived benefits to developing and 
using M&S in the program. The crux of the problem is the dilemma of costs versus risks 
and the potential return. Those who develop and use M&S need a methodology to 
evaluate investment opportunities. 
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C. RESEARCH QUESTIONS 

1. Primary Research Question 

The main question of the thesis may be stated as: “Can a rigorous business 
analysis be applied to the utilization of M&S as applied to developmental testing? 

2. Secondary Research Questions 

The underlying questions that support this overarching question are: 

• What are the background, history, policy and guidance related to using 
M&S in the DoD acquisition process? 

• What are the detailed steps to developing an M&S business case analysis? 

• How must this business case be tailored for application to developmental 
testing and what are the results when applied to available cases? 

• What conclusions and recommendations may be drawn from the preceding 
information that might be applied to M&S efforts in support of 
developmental testing? 

D. SCOPE OF THE THESIS 

The main thrust of the study will be to fully develop a business case analysis 
process for assessing the benefit of using M&S in the developmental testing community. 
Additionally, the thesis will explore, evaluate, and/or develop methodologies to assess the 
economic benefit of using M&S to enable performance of developmental testing and 
determine under what conditions M&S can be cost effective. Based on analysis, it will 
be determined if cost benefit is an effective measure or whether other extrinsic factors 
determine the worth of using M&S in performing developmental testing. 

E. METHODOLOGY 

The main thrust of the research methodology shall be to apply the developed 
rigorous business case to several developmental testing/M&S efforts. This research will 
be performed through analyzing data from studies performed about previous VPG efforts. 
This business case will be developed and refined through application to a system that was 
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tested using live testing techniques and the application of VPG M&S techniques. The 
motivation to take this approach is to show how the business case can be applied prior to 
program initiation to ensure that the economic benefit of the use of M&S in a particular 
situation can be assessed accurately. Usually M&S are applied where possible and then 
its worth is determined after completion. This technique suggests that a rigorous business 
case has not been applied in the past. Once the analyses have been performed, then the 
live testing cases will be compared to determine the economic benefit of using M&S. 
This methodology is intended to provide credibility to the business case for using M&S 
in performing developmental testing. 

F. ORGANIZATION 

This thesis will be organized in the following manner: 

• Introduction 

• M&S History, Policy and Guidance 

• Development of the Business Case Framework 

• STEP 1. Establish a Baseline 

• STEP 2. Establish a Vision and Direction 

• STEP 3. Quantify the Costs and benefits 
of Altemative/Capabilities 

• STEP 4. Evaluate Alternatives 

• STEP 5. Conduct Sensitivity Analysis 

• STEP 6. Develop a Migration Strategy 

• STEP 7. Monitor the process and Continue Feedback 

• Applying the Business Case Framework to Developmental Testing 

• Conclusions and Recommendations 

G. BENEFITS OF THE STUDY 

This thesis is timely in the sense that the DoD has created an initiative over the 

last five years to use SBA and M&S techniques throughout the acquisition process. The 

limitation of the SBA effort is that there is no clear direction or guidance for 

implementation of its techniques in a cost-effective manner. However, the acquisition 
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community has been constantly required to prove the benefits and more specifically, the 
cost benefits derived from the use of SBA. The thesis sponsor, US Army Developmental 
Test Command, will benefit from the methodology developed herein to better predict, 
assess, and account for the benefits associated with using M&S in their testing processes. 
Additionally, the Business Case Framework will provide a technique that will better 
allow documentation of M&S decisions that can withstand inquiries. This thesis will 
bring critical insights into the process necessary to determine the benefits that may be 
derived from using M&S in the testing process. This effort will add to the body of 
knowledge for the application of M&S in the acquisition community. 
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II. HISTORY OF M&S INCLUDING POLICY AND GUIDANCE 


A. MODELS AND SIMULATIONS 

Since the beginning of time, models have been used to represent abstract and 
physical phenomenon. So one should not be surprised that one of the most powerful 
tools being used by the defense acquisition community to include test and evaluation is 
modeling and simulation. Acquisition professionals are finding new, unique and 
challenging applications of models run over time creating a simulation of a system, 
subsystem, or the environment that the system is immersed in. It is indeed possible to 
create a synthetic environment where systems, both real and virtual, can be tested to 
determine the system’s proper performance for its intended mission. 

The word model brings to mind several mental images. Some remember the 
plastic pieces of an airplane, car or ship; all connected in an orderly fashion to be pulled 
apart, assembled, trimmed and sometimes painted. Others may remember watching an 
old war movie where the maneuver plans were drawn in the dirt or scale mockups used 
for mission rehearsals. A model is defined as a physical, mathematical or otherwise 
logical representation of a system, entity, phenomenon or process. A simulation can have 
two definitions: a method for implementing a model over time; and a technique for 
testing, analyzing, or training in which real-world systems are used, or where real-world 
and conceptual systems are reproduced by a model. 

The Army has used explicit representation of combat systems, combat and other 
processes for a number of years. Long before the advent of computers, the Army relied 
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heavily upon information derived from the conduct of simulations. History contains 
examples of planning and rehearsing missions using sand tables, and developing force 
structure and tactics using substitutes for weapons and weapon systems during the 
Louisiana Maneuvers of 1940. The sophistication of tools used to model and simulate 
combat systems and combat processes evolved over the years. Many major field and 
command post exercises were conducted using probability tables and the rolling of die to 
simulate the occurrence of events. Rapid advances in computer technology sped the 
evolution of M&S into the synthetic environment. 

During the evolution from predominantly physical representations, the Army’s 
use of M&S continued to support a variety of applications for five major purposes: 
education, training, and military operations; analysis; research and development; test and 
evaluation; and production and logistics. 

Since World War II, technology has advanced at an ever-increasing rate. This 
explosion of technology is moving faster than the acquisition community can acquire 
products. With technology we can accomplish what was once considered impossible. 
With the utilization of microprocessors, many new spin-off technologies are enabling 
new designs everywhere. Virtual reality with an interactive, computer-generated or 
synthetic environment is significantly changing our lives; entertainment, work, learning, 
travel and communications are all incorporating virtual reality. Today, information is 
being moved instead of people. 

B. M&S HISTORY, POLICY AND GUIDANCE 

The following will give a brief history of modeling and simulation and the policy 
and guidance that direct its use in the acquisition environment. Table 1 in Appendix A 
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contains a detailed chronological history of M&S and the contributing world technology 
events that influenced its development and application. 

Until as recently as the mid-1980s, the Army’s development and use of M&S 
were accomplished on an as-needed and as-afforded basis. In the late 1980s, the advent 
of distributed simulation technology, led by the Advanced Research Projects Agency 
(ARPA) introduced the Army to Simulation Network (SIMNET). The SIMNET, coupled 
with a downward trend in defense budgets led the Army to seek M&S application 
simultaneously addressing more than one of the purposes mentioned previously. In 
distributed simulation technology, the Army recognized the potential for linking M&S of 
various types, fidelities, and resolutions, and of establishing these linkages from 
geographically separated sites both in the Continental United States and overseas. In 
addition, the Army was assigned the role of executive agent for DoD in developing the 
technology and infrastructure to support military applications of Distributed Interactive 
Simulation (DIS). 

The Army introduced management of its models and simulations in the early 
1980s with Training and Doctrine Command (TRADOC) Regulation 5-4. Today, the 
Army executes management of its M&S through the Army Modeling and Simulation 
Office (AMSO) with policy prescribed in Army Regulation AR 5-11. Army M&S used 
in the T&E process receives its vision through the Developmental Test Command’s 
Virtual Proving Ground Master Plan. 

In June 1991, the Deputy Secretary of Defense approved a plan to strengthen the 
use of modeling and simulation (M&S). He also designated the Under Secretary of 
Defense for Acquisition and Technology (USD(A&T)), as responsible for strengthening 
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the use of M&S in joint education and training, research and development, test and 
evaluation and operation and cost analysis. In June 1992, the Institute for Defense 
Analyses (IDA) published a report titled “A Review of Study Panel Recommendation for 
Defense Modeling and Simulation.” IDA reviewed 179 recommendations made by 25 
separate study panels, over a 16-year period, concerning defense M&S. The Defense 
Modeling and Simulation Office (DMSO), using this document as a conceptual 
foundation, reviewed and classified the recommendations and set off to plan for and 
implement those that provided for new and extended applications for M&S. The DMSO 
especially focused on systems acquisition and test and evaluation. The initiative 
instituted by the Deputy Secretary of Defense is now referred to as the Defense Modeling 
and Simulation Initiative. 

Some key conclusions drawn from the IDA study were: 1) Specific areas, such as 
the architectural issues of interoperability and specification of standards, and the life- 
cycle support of defense models and simulations, deserve more attention and support; 2) 
There are many areas to which defense M&S either should be applied anew or extended, 
especially those associated with systems acquisition; 3) There are substantial needs and 
opportunities for improving management and coordination of defense M&S activities. 
Particularly of interest to this work is the fact that there is a lack of life-cycle support and 
methodologies to determine the life-cycle costs of defense models and simulations. 

Then, in June 1993, Dr. Anita Jones, the Director Defense Research and 
Engineering, established the Acquisition Task Force on Modeling and Simulation 
(ATFMS) to recommend actions which would lead to the more effective and integrated 
use of modeling and simulation throughout the acquisition process. The final report of 
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the ATFMS, dated June 1994, concludes that 1) the effective, integrated use of M&S in 
the acquisition process is being impeded by the lack of an overall M&S architecture; 2) 
the lack of this M&S architecture has led to not getting the most out of the investment 
that is being made in M&S for acquisition; 3) clearly designated leadership and an 
appropriate coordinating mechanism are required to stimulate progress in the creative 
application of M&S to the acquisition process; 4) additional education and training on the 
capabilities and limitations of M&S are required for all participants in the acquisition 
process; and 5) it is time to apply advanced M&S and related tools to enhance real 
acquisition programs. (Parker, 1994) The recommendation of the ATFMS to develop an 
overall M&S architecture spurred the initiation of the development of the High Level 
Architecture (HLA) within the DoD. 

Beginning in 1994, the Army implemented a policy that requires all Army 
acquisition strategies for ACAT I and II programs, subsequently expanded to Advanced 
Technology Demonstrations (ATD), to include a Simulation Support Plan (SSP). In this 
plan, the PM must lay out the functional requirements for M&S to support engineering 
and combat developments, test and evaluation, training, and military exercises to support 
the program. The PM must also develop an M&S acquisition strategy identifying 
resources required bringing the M&S to fruition. 

Through the Army M&S Management process M&S standards categories have 
been developed to enable the use of M&S throughout the three Army M&S domains; 
Training, Exercises and Military Operations (TEMO), Advanced Concepts and 
Requirements (ACR), and Research, Development and Acquisition (RD&A). M&S that 
are developed at scientific and engineering levels are included in the RD&A domain. 
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Developmental Test and Evaluation using M&S falls within the RD&A domain. 
According to AR 5-11, Management of Army Models and Simulations, M&S developed 
for use in Test and Evaluation will be managed over their life cycles in accordance with 
the responsibilities defined in AR 70-1 and AR 73-1. (AR 5-11, 1997) Within the Army 
Standards Categories, there is a category called Cost Representation. This category 
includes the data, tools, algorithms and techniques necessary for accurately costing and 
consistently portraying all aspects of activities portrayed in models and simulations. The 
Cost Representation standards category currently lists two requirements: develop 
methods to cost all elements portrayed in M&S and standardize techniques for comparing 
costs of alternatives. To date however, this standards category group has not published 
anything to address their two listed categories. This group should be followed for future 
cost representation information. This group is lead by the Director of the United States 
Audit Agency (USAA) Cost and Economic Analysis Center, Ruth Johnson. 

On Sep 10, 1996, Paul Kaminski signed out the DoD High Level Architecture 
(HLA) for Simulations Policy which designates the HLA as the standard technical 
architecture for all DoD simulations. (Kaminski, 1996) The HLA has been designated as 
the standard technical architecture for all DoD simulations. This mandate for HLA 
compliance supersedes all previous requirements for DoD simulations to comply with 
other simulation standards such as DIS or Aggregate-Level Simulation Protocol. All 
Army simulations will meet DoD standards. HLA establishes a common high-level 
simulation architecture to facilitate the interoperability of all types of simulations among 
themselves and with Command, Control, Communications and Intelligence systems. The 
HLA will also facilitate the reuse of M&S as the Army moves into an era of federations 


14 



of simulations producing synthetic battle environments across all domains and inclusion 
of M&S into all major acquisition programs. 
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III. DEVELOPMENT OF THE BUSINESS CASE FRAMEWORK 


A. RESULTS OF SURVEYS 

A survey conducted in 1999 by an ICAF team indicated that PMs are investing in 
M&S to support program development; however, “most of the investment decisions were 
based on intuition or need-based factors. Most decisions were made without detailed 
quantitative analysis (Brown et al., 2000)”. For the most part, PMs accepted modest 
investment in M&S because they believed that such investments would benefit the 
program. Most of these efforts, however commendable, usually lacked the use of a 
methodical cost/benefit analysis. “The lack of a structured business case analysis made it 
difficult, if not impossible, for PMs to articulate or substantiate their investment strategy” 
(Brown et al., 2000). In some cases PMs developed and explored M&S capabilities 
without even knowing what benefits might be achieved. Often the PM is willing to take a 
risk in using M&S in the development of his program with the hopes that a marginal yet 
unquantified benefit will be realized. The PMs in the ICAF study were “embracing new 
technology; however, without a sound business case approach, they could not really 
assess if the investment would realize considerable benefits or generate prohibitive costs” 
(Brown et al., 2000). Most PMs justified their M&S investment based on one or more of 
the following: reducing design cycle time; augmenting or replacing physical tests; 
helping resolve limitations of funds, assets or schedules; or providing insight into issues 
that were impossible or impracticable to examine in other ways. (Brown et al., 2000) 

Much of the feedback from the ICAF study reflected that PMs had tried to 
examine and measure the costs and benefits of using M&S but without the aid of any 
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business case analysis format. Of the few reported to have used a disciplined approach, 
approaches and results varied among PMs surveyed. This use of inconsistent 
applications or approaches leads to mixed results that cannot be readily compared or 
evaluated. ‘The majority of respondents suggested the question of using M&S was not 
one of “Should I” but rather, “How can I?” This demonstrates general acceptance that 
M&S is required in an efficiently managed program, but it suggests the question, “At 
what cost?”’ (Brown et al., 2000) In the cases where the cost investment is prohibitively 
expensive or the long-term benefits are marginal, the decision-maker should select from 
alternatives, be it “foregoing M&S, partnership with other PMs to share costs, or 
leveraging the investment of others. This is an important decision, given the DoD’s 
current fiscal climate.” (Brown et al., 2000) 

PMs want “confirmation that investment in M&S will yield direct savings within 
their budgets.” (Brown et al., 2000) Considering that much of the benefit of M&S 
investment is intangible, traditional cost benefit measurement approaches may not 
provide an accurate assessment. In this situation, PMs may dismiss using M&S tools if 
the estimated ROI is not significant for their immediate project needs. However, since 
some projects are relatively short-lived, the longer-term residual benefits may very well 
be missed due to short sightedness on the part of PMs. While traditionally the benefits of 
M&S tend to be discussed in terms of ROI, several alternatives for business case analysis 
can just as effectively justify M&S investments. “The challenge is to define an 
appropriate strategy and priorities to address the business value proposition.” (Brown et 
al., 2000) 
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A disciplined business case approach and methodology has many benefits. “It 
can help bring the aggregate benefits into focus and strengthen the argument for M&S 
investment.” (Brown et al., 2000) A business case analysis provides a convenient 
mechanism for project management to explore investment possibilities and assess the 
potential risks. It can be an easy-to-follow, logical thread that lays the groundwork for 
current and future programs to attain useful information, metrics and measures. “This 
approach forces a timeline, captures benefits and enables authorities to decide if the 
return is worth the investment in simulation technologies.” (Brown et al., 2000) 

B. MAKING THE BUSINESS CASE 

The business case entails addressing seven basic areas in the prescribed order. 

• Establish a Baseline 

• Establish a Vision and Direction 

• Quantify the Costs and Benefits of Altemative/Capabilities 

• Evaluate Alternatives 

• Conduct Sensitivity Analysis 

• Develop a Migration Strategy 

• Monitor the Process and Continue Feedback 

(Brown et al., 2000) 

Corporate America is taking a methodical approach to investment decisions 
regarding everything to include M&S. Although these methodological approaches are 
“primarily profit driven, they share common maxims of production, such as cost 
reduction, efficiency and cost avoidance, to that of DoD.” (Brown et al., 2000) 
Essentially, a business case will assist the decision-maker in evaluating which of a 
number of logical alternatives will best meet the objective. 
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1. Step 1. Establishing a Baseline 

The first step in this decision-making process is to establish an accurate baseline. 
The key in determining the success or failure of a project, program or function is 
establishing its current state before any changes are considered or implemented. “The 
baseline provides a benchmark from which decisions will be weighed and assessed.” 
(Brown et al., 2000) Performance measures, regardless of type, must have a baseline 
measurement to show the changes/improvements a project undergoes as it achieves it 
goal. Establishing a baseline measure is essential in establishing the validity of a 
performance measure. A baseline measure is usually the first measure taken of a system 
or project. Subsequent measurement may be defined as the new baseline if substantial 
changes to the system make the earlier baseline obsolete or reduce the effectiveness of 
the particular performance measure. “The baseline should include a clear enunciation of 
assumptions and constraints.” (Brown et al., 2000) 

Assumptions are explicit statements describing the present and future 
environment. They reduce complex situations into manageable proportions. These 
assumptions normally provide some indication of the estimated future workload, the 
useful life of the investment or system, and the period of time over which alternatives 
will be compared. Assumptions should also discuss sunk costs and realized benefits, but 
are not included as part of the baseline. Constraints are those factors that limit 
alternatives. Normally they are expressed in terms of time, finances, institutional or 
regulatory statutes or directives and physical plant/assets. 

The baseline must identify the high value portions of a program in order to 
evaluate the appropriateness of various alternatives. In other words, the decision-maker 
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must look at the areas of the program which will take the largest portion of program time 
in which to invest M&S resources to achieve a time benefit. Additionally, the decision¬ 
maker could also address which areas of the program are most costly and then 
appropriately apply M&S resources to potentially achieve a cost savings/benefit. 
Following an informed examination of the program being analyzed, the PM would be 
better able to allocate M&S spending where it would offer the greatest potential savings 
or benefits. The baseline must determine these high value areas for an effective business 
case analysis to be performed. These program specifics form the drivers to the program, 
which in turn drive the investment process. (Brown et al., 2000) 

2. Step 2. Establishing Vision and Direction 

The decision-maker “must look to the future, then bridge the gap between present 
knowledge and that required to make future products a reality.” (Brown et ah, 2000) The 
acquisition decision-maker or PM must establish the program's vision for using M&S. 
As the PM is developing his plan for use of M&S in support of his acquisition program, 
he should keep focused to ensure success. The PM should use the list of probing 
questions contained in the SB A “Cheat Sheet” which is provided in Appendix B. A 
performance plan needs to be established for the program that should contain 
performance goals and measures. Performance goals should be expressed in an objective, 
quantifiable, and measurable form. Performance measures should be established that are 
indicators for measuring or assessing the relevant service levels, outcomes or outputs and 
comparing actual program results with the established performance goals. 

The performance plan for the program must identify the performance goals and 
measures that are based on the goals and objectives of the agency’s strategic plan. 
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Performance goals that are representative of milestones in the achieving the long-term 
objectives contained in the organizational strategic plan should be included in the 
performance plan. The performance plan should establish the performance measures or 
indicators to be pursued in measuring or assessing the relevant service levels, outcomes 
or outputs and comparing actual program results with the established performance goals. 
Additionally, the performance plan will describe the operational processes, skills and 
technology, and the human, capital, information, or other resources required to meet the 
performance goals. The performance plan will provide a basis for comparing actual 
program results with the established performance goals; and describe the means to be 
pursued to verily and validate the measured values. 

For some programs, there may be a gap between the capabilities provided by 
existing resources, including M&S resources and program goals/objectives as stated in 
program performance plans. Some performance gaps may be resolved by reengineering 
processes with or without the use of M&S. If reengineering of a process is needed it 
should be done prior to determining what M&S investments are needed to support the 
redesigned process. 

A functional requirement analysis should be performed to determine the 
requirements that the M&S investment must meet to fill the performance gap. The 
analysis should identify: the performance criteria, goal, or ultimate output, a definition of 
the common uses of the M&S investment, a ranking of the requirements in order of 
importance, and a decomposition of functional requirements into self-contained features 
(GSA, 1998). To allow flexibility in evaluating various solutions, functional 
requirements should not be described in equipment and software terms, but in terms of: 
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business outcome, mission, purpose, capability, organizational program components 
involved, schedule and cost objectives, and operating constraints. The m inimum 
information that should be identified by an analysis of functional requirements for an 
M&S system includes: system functional description, inputs/outputs, processes, data 
characteristics, performance criteria (accuracy, validation, timing, flexibility), interfaces, 
failure contingencies and security specifications. 

Wherever possible, requirements for M&S systems should be stated using an open 
system architecture. The open system architecture should encompass the following 
characteristics: user applications not tied to a single hardware or system software 
manufacturer, new functionality added from different contracts without significant effort, 
other systems tied into or interfaced with the system without significant effort, and the 
system fits the organizational M&S architecture plan. Other agencies that may have 
acquired systems to accomplish similar goals should be identified and management 
should look for cross-agency, other M&S domain solutions or Government wide 
economies to avoid duplication of effort. 

As with other developing technologies, “the vision should consider how M&S 
tools can improve program costs, scheduling and performance, and whether scientific 
knowledge exists to support such M&S investment.” (Brown et ah, 2000) The vision 
should drive or focus what the M&S tools should be trying to solve, not the other way 
around. The vision statement clearly identifies what M&S capabilities are to be 
achieved, without dictating how they will be accomplished. 

During this program vision/direction planning effort, determining the functional 

requirements, feasibility, alternative, cost, and benefits must be accomplished as 
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efficiently as possible. An Integrated Product Team (IPT) should be formed to help 
ensure the program’s success throughout its life cycle. The IPT will assist the program 
manager in completing and documenting the analyses needed for each project life cycle 
phase. The IPT for projects involving procurements must include procurement 
specialists, financial analysts, M&S experts and weapon system experts. The IPT will 
help identify functional requirements that describe functional and customer needs that 
must be satisfied by the M&S investment. The IPT must perform market research to 
ascertain if the market can provide the desired models and assets necessary to meet the 
program requirements. This market research/analysis is designed to produce a list of 
alternatives, with accompanying data necessary to assess affordability, benefits and costs. 

Market research is done before the development of any formal requirements 
documents are developed. The following information should be collected during the 
initial market survey: 1) Availability of commercial items or other Government assets 
available to meet the need, and whether they might require modification; 2) The 
customary practices and the associated costs for customizing/modifying items to meet the 
need; 3) The customary practices regarding warranties and discounts for the identified 
products; 4) The laws and regulations which may apply to the acquisition of identified 
products; and 5) The distribution and support capabilities of possible suppliers. The 
decision on the contract type should be taken from the results of the market research. 
Once the vision and direction have been set and the requirements have been drafted, the 
cost and benefits of each alternative discovered during the market research must be 
quantified. 
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3. Step 3. Quantify the Costs and Benefits of Alternatives/Capabilities 

a. Introduction 

Since alternatives are logical packages of initiatives that work well 
together (Kidwell, 1998), one alternative is the status quo. This alternative is often 
identified during the development of the baseline in Step 1. In some cases, the baseline, 
often a physical prototype or live test, can be more cost effective than the use of M&S 
depending on the configuration or technology employed. “Other alternatives should 
represent various combinations of M&S tools that help achieve the vision.” (Brown et al., 
2000) 

In the conceptual stages of a project, some of the detailed information 
about benefits and costs associated with different alternatives may not be available. For 
this reason, the benefit and cost analysis information should be updated and corrected as 
necessary as the project planning proceeds to its later phases and more information 
becomes available. 

In determining alternatives, the decision-maker must consider both the 
immediate and long-term effects of each proposal. “First, what does the program need in 
order to perform better? Second, what does the program need to enable its survival until 
the next stage?” (Brown et al., 2000) Ultimately, a program must satisfy all requirements 
while at the same time, it must endure and survive each step of the process without being 
cut. “It does no good for a program to bankrupt itself with massive unfocused M&S 
investment early on.” (Brown et al., 2000) Each investment must produce value during a 
timeframe that is appropriate for the program to achieve its performance goals. In order 
to survive, the PM must meet his near-term milestones in order to survive; therefore, he 
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must create a program that can withstand the tests of time. The alternatives must also 
consider the technological advances in M&S tools that might occur. “A program might 
not be justified in spending huge sums of money on M&S technology that may be 
superceded and rendered obsolete in 2-5 years.” (Brown et al., 2000) 

The decision-maker when performing a cost benefit analysis must identify 
all costs that are incident to achieving each alternative. M&S can be expensive to 
develop, particularly in domains where the scientific principles are not fully understood 
as applied to the problem. While additional research can fill the knowledge voids, the 
cost of this research must be factored into the analysis of alternatives. These should 
include the opportunity costs of assets and resources, which are the alternative value 
foregone when an asset is used for other purposes. (DoDI 7041.3, 1995) They also 
include non-recurring and recurring costs. Life cycle costs should include all costs, non¬ 
recurring and recurring that occur over the life of an alternative. (GSA, 1998) 

b. Benefit Cost Analysis (BCA) Framework 

Consistent with Office of Management and Budget (OMB) Circular A-94 
guidance, a BCA should encompass and address the following elements: explicit 
underlying assumptions used to arrive at the estimates of future benefits and costs, 
evaluation of alternative means for achieving program objectives, and plans for periodic, 
results oriented evaluation of the actual costs, benefits, and program effectiveness 
attributable to the investment. 

c. BCA steps 

The BCA process encompasses the following steps: (1) Identify 
assumptions and constraints, (2) Identify alternatives and their schedules, costs and 
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benefits, (3) Evaluate alternatives, (4) Perform risk and sensitivity analysis, (5) Develop 
performance goals and measures for monitoring the project. 

The first two steps of the BCA process apply to this step of the business 
case analysis. The rest of the BCA steps will be applied to the latter steps of the business 
case analysis. 

(1) Assumptions and Constraints. Assumptions are explicit 
statements used to describe the present and future environment upon which the 
benefit/cost analysis is based. The purpose of assumptions is to reduce complex 
situations to problems of manageable proportions. 

OMB Circular A-94 requires analyses to be explicit about 
underlying assumptions to arrive at estimates of future benefits and costs and include a 
statement of the assumptions, the rationale behind them, and a review of their strengths 
and weaknesses. Examples of assumptions include estimated future workload, estimated 
useful life of an investment or system, and the period of time over which alternatives will 
be compared. Constraints are factors external to the relevant environment that limits 
alternatives to problem resolution. They may be physical, time related, financial, or 
institutional/regulatory. They provide boundary limits for the alternative solutions to a 
particular problem. 

(2) Identifying and Estimating the Benefits and Costs of 
Alternatives. Some examples of simulation related alternatives are: do nothing, use 
Commercial Off the Shelf (COTS) packages or purchase new software or equipment, 
modify existing hardware/software, develop new software, and purchase services. 
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One alternative that should always be considered is continuing the 
status quo. Each alternative will have its own mix of resources. Costs must be identified 
and itemized at a level of detail consistent with the budgeting process. Alternatives will 
also have different benefits realization periods and some additional or direct benefits. As 
the project or procurement process proceeds, the BCA and budget requests will be 
updated to reflect the most current information on alternatives based on the project and 
procurement progress. 

The estimate of costs and benefits of an investment or project 
should show the difference that results from making the investment; specifically the 
change in cash flows as a result of undertaking the project. Basic questions to be asked 
are: 

• What additional funds will be required to carry out the chosen alternative? 

• What additional revenues will be created over and above any existing 
ones? 

• What costs will be added or removed as a result of the investment? 

Benefit-cost analysis for M&S investments compares the costs of 
the M&S investment or project (whether it be a new system, a replacement system, 
system enhancement, or a hardware/software purchase) to the savings derived from the 
expected business and operational improvements resulting from the M&S investment or 
project. The basic elements of cost comparison are the total M&S investment/system and 
business costs if the system is implemented versus the total system and business costs if 
the system were not implemented or if the current system is continued. The savings 
resulting from the system implementation are associated with tangible benefits. 
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Additional intangible benefits are also documented and considered in the decision to 
approve system development. 

(3) Tangible and Intangible Benefits and Costs. Consistent 
with OMB Circular A-94, both tangible and intangible benefits and costs should be 
recognized. The relevant cost concept is broader than private-sector production and 
compliance costs or Government cash expenditures. Costs should reflect the opportunity 
costs of any resources used, measured by the return to those resources in their most 
productive application elsewhere. A-94 provides additional guidance on identifying and 
measuring benefits and costs including: incremental benefits and costs and sunk costs, 
transfers, indirect measures of benefits and costs, multiplier effects, treatment of 
inflation, discount rates, and lease purchase analysis. 

After the decision has been made that it is beneficial for an agency 
to acquire the use of a capital asset, OMB Circular A-94 guidelines should be used to 
perform a Lease-Purchase analysis to determine if the agency should purchase or lease 
the asset. Lease-purchase analyses should compare the net discounted present value of 
the life cycle cost of leasing with the full costs of buying or constructing an identical 
asset. 

(4) Identifying and Estimating Costs. When considering the 
costs of projects/altematives, one must take into consideration the project’s total life 
cycle cost, as defined in OMB Circulars A-94 and A-109 to include all acquisition costs 
and the cost of operations. 

(5) Project Life. It is often difficult to estimate the life of a 
project. The accepted criterion is the continued ability to generate satisfactory cash flows 
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or other intangible benefits. The period of time over which the savings or benefits to be 
gained from a project may be expected to accrue is the economic life of a project. The 
economic life is generally the lesser of physical life, technological life or 
mission/product-market life. The mission life is that period of time over which a need for 
the asset or program is anticipated. The physical life is the period during which a facility 
or piece of equipment will be available for use. The technological life is the period of 
time before which improved technology would make an asset obsolete. The project life 
of the M&S investments sometime occur several years prior to the time the project starts 
providing benefits. This elapsed time period between initial funding and the 
commencement of the economic life is referred to as lead-time. Project life consists of 
the total of the lead-time and the economic life. 

(6) Methods of Alternative Comparison. Economic lives and 
lead times can vary among alternatives. The following guidelines are recommended for 
determining the comparison period. If both the economic lives and lead times for all 
alternatives are the same, there is no problem as the comparison will be between the same 
project life. In the case where the same economic lives and different lead times, the first 
year that expenditures must be made for any one of the alternatives, should be considered 
the base year or first year for all the alternatives. When dealing with different economic 
lives, one method is to let the economic life of the dominant asset prevail with subsidiary 
assets replaced us necessary. Another method when dealing with different economics 
lives is to use the shortest economic life and impute residual value in the asset with the 
longer life. Because of the inherent uncertainties of making estimates in distant years, in 
some cases it may be necessary to set arbitrary limits on the planning horizon to be used 
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in the analysis. This planning horizon can be shorter than the estimated economic life of 
the project. 


(7) Cost categories. Cost categories to consider when 
estimating the cost of an altemative/project include the categories following, which are 
not mutually exclusive. 

• Research and development: These costs are often associated with the 
development of new simulation systems and include items such as model 
development or test bed costs. 

• Investment: Investment costs are essentially one-time costs and include 
costs such as: land, new construction, rehabilitation, equipment, software 
purchases, system development (functional requirements, design, analysis, 
programming, testing, conversion), and relocation. 

• One-time personnel cost (recruitment, separation, training, travel, etc.) 

• Value of Existing Assets Employed: This is the value of existing assets. 
This value is included in the investment cost only when the existing asset 
is currently in use on some other project, or was intended for sale. 

• Terminal/residual value: This is the expected value of buildings, 
equipment or other assets at the end of their economic lives and is treated 
as a reduction in the life cycle cost of the particular alternative for which 
the use of the asset is intended. Residual value is the computed value of 
assets as any point in time. Residual value may or may not coincide with 
terminal value. Terminal/residual value should be applied to existing asset 
replaced as well as new assets being acquired. 

• Operation and Maintenance Costs: These costs occur continually over the 
useful life of the project. They include labor costs of operating and 
maintenance personnel, fuel and power costs, operating and maintenance 
supply costs, spare and repair parts costs, insurance costs, taxes, and a 
share of indirect (overhead/burden) costs. These costs can be substantial 
and occur over time until the structure, system or equipment is retired 
from service. 

• Variable costs: These are a group of costs that vary in some relationship to 
the level of operational activity (such as direct labor, direct material) 

• Fixed costs: These are a group of costs that do not vary with output. 

• Total cost: This is the sum of all life cycle costs associated with the 
product/system. 

• Unit cost: This is the total cost divided by some related base and may be 
expressed in terms of cost per item produced, per person, etc. Unit cost 
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represents an average that may change with the magnitude of the 
numerator, denominator, or both. 

• Recurring and nonrecurring costs. Recurring costs are those costs that 
occur from one period to the next at specified intervals; whereas, non¬ 
recurring costs are one-time, non-repetitive costs. Life cycle costs 
embrace all costs, non-recurring and recurring that occur over the life of 
an alternative. 

• Sunk Cost: These are costs that have already been incurred and cannot be 
recovered or altered by future action. They are irrelevant to the benefit- 
cost analysis because only the future consequences of investment 
alternatives can be affected by current decisions. 

For the more avid reader, the definitions of the cost factors that 
apply to these cost categories can be found in Appendix C. 

(8) Cost Estimating Techniques. A thoroughly reasoned 
benefit-cost analysis requires the collection of financial information called cost elements 
from budget documents along with estimates of proposed simulation investment/system 
costs. The selection of cost estimating techniques depends on the amount and detail of 
available data and the time and resources available to develop the cost. The required 
level of effort for the different estimating techniques ranges from extreme analytical 
detail to intuition. The three cost estimating techniques are the bottom-up method, 
parametric analysis method and the analogy method. 

The industrial engineering/bottom-up method consolidates 
estimates from several separate work segments into a total project estimate. It involves 
segmenting the total product into single parts for which detailed estimates can be 
established. Where detailed data exists, the industrial engineering approach can result in 
extremely detailed and complete estimates. 

The parametric cost estimating or parametric analysis method 
focuses on what the project is supposed to accomplish or yield compared to similar 
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projects. The yield or benefits form the basis or parameters for the cost estimates. Once 
these benefits and their measures are established, relationships between the parameters 
and their costs are developed, mainly form historical data. This method is used when 
data is inadequate for employing the industrial engineering approach. It is also a 
preferred method for deriving cost estimates at the earliest stages of development. 

The analogy method uses judgment, specifically analogies, which 
are direct comparisons with similar, historical systems or products. This method requires 
expertise and intuitive reasoning. When little historical data is available and neither the 
industrial engineering nor parametric methods can be used, the analogy method is used 
but it is seldom the most accurate. 

In developing cost data for a life cycle cost analysis, one should 
initially investigate possible data sources to determine what is available for direct 
application to analysis objectives. If the required data are not available, the use of 
parametric, cost estimating techniques may be appropriate. Existing data banks, initial 
system planning data, supplier documentation, reliability and maintainability predictions, 
test data should all be investigated a potential data sources. 

(9) Identifying and estimating benefits. All benefits resulting 
from each alternative should be identified. Both quantifiable and non-quantifiable 
benefits should be identified and described. Determining benefits is the most difficult 
part of the benefit/cost analysis because it is often difficult to identify all benefits and 
accurately quantify and monetize them. OMB Circular A-94 suggests the principle of 
willingness-to-pay to obtain a given benefit and that market prices are a good place to 
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start. To the extent possible, benefits should be expressed in quantifiable terms and 
clearly linked to the program goals and needs identified in previous planning stages. 

Most benefits will be in terms of improvements in effectiveness, 
efficiency, or customer satisfaction. Example areas of benefits include: operating 
efficiency, reliability/maintainability, accuracy manageability, availability service life, 
quality ecology, economy morale, safety security, and regulatory compliance. Examples 
of types of quantifiable and monetizable benefits are: reduced resource requirements 
(such as support services, supplies, personnel, training, lease, rental, maintenance, 
computers), improved data entry (resulting in reduced staff time, lowered error rates), 
improved operational effectiveness (resulting in reduced error rates, improved timeliness, 
increased productivity, better quality products), cost avoidance (by eliminating future 
staff growth, minimizing penalties for delays, elimination additional equipment 
requirements). 

d. Benefit-Cost Analysis Framework. 

The decision to undertake a simulation investment or project is based on 
the assumption that the business improvements resulting from the system exceed the 
costs of modifying business operations and maintaining the current simulation or system 
(if it exists). Benefit-cost analysis makes explicit the assumed business rationale that 
justifies investments in simulation versus live systems. Benefit-cost analysis has four 
major elements: total business and system costs with the simulation investment/new 
system, total business costs without the simulation investment/new system, tangible 
benefits, and intangible benefits. 
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The determination of costs and tangible benefits is based upon five basic 


cost elements: 


• Business costs with the simulation investment new system—The total costs 
to carry out the business functions and processes to be automated by the 
simulation investment/system. 

• Business costs without the simulation investment/new system—The total 
costs that would be incurred to continue the business functions and 
processes with the current level of automation (which may be no 
automation). 

• Nonrecurring costs of the new system—One time expenditures that will be 
incurred in the design, development or acquisition, and implementation of 
the new system. These expenses will not be incurred after a system is 
operational. 

• Recurring costs of the simulation investment/new system—Ongoing 
expenses that will continue throughout the investment’s/system's life 
cycle. Most of these costs will be incurred during the operational phase of 
the system. 

• Costs to continue the current system (if there is one) -The expenditures 
that would be made if the organization continued to operate the existing 
system (these may include recurring and non-recurring costs). 

Business costs are presented as a total budget projection for the business 

operations affected by the proposed simulation investment/system. Analysis of business 

costs should consider the same factors that are applied in developing multi-year budget 

projections. The estimates for the benefit-cost analysis should be comparable to those 

produced in other budget exercises. 


Costs of a simulation investment/system are the costs required to design, 
acquire, develop, implement, and operate the simulation investment/system. These are 
the costs related to the simulation/system itself and not the business functions supported 
by the simulation/system. Costs include both business and system costs with and without 
the simulation investment/system. This cost comparison quantifies the financial impacts 
of a "go" or "no go" decision. 
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The cost of operating the business without the simulation/system 
highlights the investment managers would be forced to make in maintaining current 
business practices and system operations. The cost of operating the business with the 
new or enhanced system highlights the tangible bottom line payoff of the proposed 
system. Total costs with the simulation investment or new information system will, in 
most cases, be more than continuing current operations. Savings can accrue in the 
business operations that exceed the additional costs associated with design, development, 
acquisition, and maintenance of the simulation investment/information system itself over 
a projected life cycle. 

Tangible benefits can be measured as specific cost savings to the 
organization. Tangible benefits are the cost savings resulting from changes in business 
and system operations. Each item in the cost analysis that has a projected saving must be 
associated with an operational change that will produce the reduction in projected 
expenditures. For example, the cost of continuing operations without the simulation 
investment/new system may include the hiring of additional technical staff to continue to 
manually process projected increases in workload. If the proposed simulation 
investment/system were implemented, technology would replace these manual processes 
and no additional personnel would be hired. 

Intangible benefits are difficult to measure in financial terms. Despite 
their lack of financial rationale, they may be sufficient to justify the system independent 
of cost. In the Federal environment, compliance with legal and regulatory requirements 
is an intangible benefit that can, on its own, justify the investment in information and 
simulation systems. Other examples of intangible benefits are improved customer 
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satisfaction, faster service, and increased employee job satisfaction. Intangible benefits 
must be supported by a clear link to specific outcomes of system implementation. 

(1) Benefit-Cost Model Components. This section presents a 
discussion of the information collection requirements for the proposed benefit-cost 
model. It is organized around identifying fully allocated current simulation/system costs 
and proposed project expenses. Growth values such as present value calculations, cost of 
money and interest expense are factored into this model. The model is expressed as 
simply as possible and is consistent with Federal guidelines governing benefit-cost 
analysis. 

Information collection is intended to reflect a mixture of cost 
projections and assumptions that system owners have gained through operating 
experience. Information concerning new proposed systems should be organized and 
recorded on the project worksheet (Figure 1. Benefit-Cost Calculations). Backup 
documentation to this cost worksheet should be explained in a narrative form sufficient to 
clarify assumptions about the numbers. Complete information describing the current 
systems should also be provided along with any foreseen benefits from continuing current 
operations. 
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CALCULATION 

FORMULA 

RESULT 

Business Savings 

Business Costs Business Costs Present 

Without the - With The New = Business x Value = 

New M&S System M&S System Savings Factor 

(a) Present 

Value of 

Business 

Savings 

Net System Costs 

Recurring + Costs of Net Present 

Non-Recurring - Continuing = System x Value = 

Costs of New Old System Costs Factor 

M&S System 

(b) Present 

Value of Net 
System Costs 

Benefits 

Present Present Value 

Value of - of Net M&S System = 

Business Costs (b) 

Savings (a) 

(c) Net Present 
Value of 

Benefits 

Benefit-Cost Ratio 

Present Present 

Value Value of = 

of Net System 

Benefits (c) Costs (b) 

Present Value 
Benefit-Cost 

Ratio 


Figure 1. Benefit-Cost Calculations. (GSA, 2000) 

The model is designed to allow comparison of costs with and 
without the simulation investment/new system: non-recurring costs, recurring costs, 
residual values and benefits over an estimated systems life. Each benefit-cost worksheet 
should be carefully analyzed to ensure completeness in capturing fully allocated cost 
projections. 


The cost worksheet should be completed for the full life cycle of 
the proposed simulation project/system. The life cycle includes design, acquisition, 
development, implementation, maintenance, and disposal. Supporting documentation 
should identify the expected time period for each life cycle stage of the system. A six 
year system life cycle is assumed in this methodology. Six years reflects the impact of 
rapidly changing simulation technology on the useful life of systems. Some 
simulation/systems may have a longer or shorter life cycle. In these instances the benefit- 
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cost analysis should include the rationale for the life cycle. The methodology and 
supporting worksheets can be adjusted to fit the projected system life cycle. 

(2) Business Operation Costs. This category of cost elements 
identifies the total costs to carry out the business functions first with the current level of 
automation and then when they are automated by the simulation investment/new system. 
The costs need to be calculated both with and without the proposed new simulation 
system. Business operations are the activities and resources used to conduct the functions 
to be supported by the proposed simulation system. Business operation costs are defined 
by the following cost factors: personnel salaries and fringe benefit, supplies/equipment, 
facility space occupancy, utilities, maintenance, travel, training, incidentals, interagency, 
and other identified costs. 

(3) Nonrecurring Costs of the Simulation Investment/New 
System. This category of cost elements identifies the one-time expenditures that are 
incurred in the design, acquisition, development, and implementation of the new 
simulation system. Nonrecurring costs of the new system are defined by the following 
cost factors: conversion costs, replacement or upgrade systems, hardware, software, 
communications, contracting, travel, training, studies, parallel operations, incidental 
expenses, residual value, and other identified costs. 

(4) Recurring Costs of the Simulation Investment/System. 
This category of cost elements identifies the ongoing expenses that are incurred to 
maintain and operate the simulation/system after implementation is completed. Costs 
should be projected for the entire useful life of the system: parallel operations, hardware 
lease or rental, software lease or rental, communications, other equipment, facility space 
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occupancy, utilities, maintenance supplies, personnel salaries and fringe benefits, 
security, travel, training, system testing and back-up, incidentals, interagency, and other 
identified costs. 

(5) Costs of Continuing the Existing Simulation System. This 
category of cost elements identifies the simulation system costs that would be incurred if 
the existing level of automation or simulation were continued instead of developing the 
new system. The costs of continuing the old simulation/system are defined by the 
following cost factors: hardware lease or rental, software lease or rental, communications, 
other equipment, facility space occupancy, utilities, maintenance, supplies, personnel 
salaries and fringe benefits, security, travel, training, system testing and back-up, 
incidentals, interagency, and other identified costs. 

(6) Calculated Values. In order to compare costs and benefits 
at a point in time, present value tables are used in the model. OMB Circular A-94 defines 
the standard criterion for deciding whether a Government program can be justified on 
economic principles as the net present value of benefits. Net present value is the 
projected savings resulting from the program reduced by the net investment required 
developing and implementing the program. Using the costs of business operations and 
systems described above, five basic calculations are made to assess the benefits that will 
result from the new system. These calculations are described below and in Figure 1. 

Business Savings. Business savings is the difference between the 
cost of business operations without the simulation investment/new system and the cost of 
business operations with the new system. The result of this calculation is the business 
savings of the new system. 
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Net System Costs. Net system costs is the difference between the 
cost to continue to operate the old simulation/system (if there is some existing 
simulation/system) and the cost to design, acquire, develop, implement and maintain the 
simulation investment/new system. Net simulation/system costs are the additional 
investment that the organization will make in the simulation investment/new system or, if 
the simulation investment/new system is less costly, the system-related savings that will 
result from the new system. 

Present Value Factor. Cost information is adjusted in the model 
using tables to multiply costs and benefits by discount factors consistent with OMB 
Circular A-94 guidance to determine the present value based on the year of occurrence. 
Note: OMB Circular A-94 is updated periodically. Those conducting a BCA should 
ensure that they are using the proper discount rates, consistent with the latest version of 
OMB Circular A-94, by checking the Internet at 
http://www.whitehouse.gov/WH/EOP/OMB/html/circulars/. 

Net Present Value (Of Benefits). Net Present Value is the business 
savings increased or decreased by the net simulation/system costs. If the new system 
reduces system costs, tangible benefits will be higher than business savings. If the new 
system represents increased costs, tangible benefits will be less than the business savings 
of the new system. 

Benefit-Cost Ratio. The Benefit-Cost Ratio is the value of the 
tangible benefits compared to the net simulation/system costs. Benefits will generally 
exceed costs and this ratio will almost always be greater than 1. If a system has a benefit- 
cost ratio less than 1, it must be entirely justified by its intangible benefits. 
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(7) Benefits. New simulation systems provide opportunities 
for a broad range of improvements to technical business operations. Not all benefits of 
automation will result in a tangible benefit that reduces costs. Intangible benefits can be 
an important factor in deciding to proceed with the development of a simulation system. 
Intangible benefits should be documented as part of the benefit-cost analysis and included 
in the narrative that describes the proposed system. Intangible benefits should be 
considered with the benefit-cost ratio for determining the rationale for continuing with 
the proposed system. In identifying tangible and intangible benefits of the new system, 
the following should be considered: 

Reliability Improvements. The benefit gained in the reduced risk 
of system malfunction or failure, and reduced downtime for manual operations versus a 
comparison system, for performing the same or equivalent tasks. 

Accuracy Improvements. The benefit gained in process 
simplification and streamlining. Ease of data input and accuracy rates that reduce overall 
errors are reported here. 

Labor Productivity Improvements. The benefit gained in 
performing the same functions and tasks for fewer hours of personnel time. These 
improvements may allow staff to work on other activities, but do not result in an actual 
reduction in personnel. 

Grade of Service Productivity Improvements. The benefit gained 
in performing a service more efficiently or effectively to the direct benefit of the 
warfighter or taxpayers. 
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Compliance With Legal And Regulatory Requirements. The 
benefit gained by meeting procedural or performance guidelines specified in laws and 
regulations. Physical environmental limitations can often be overcome by stimulating the 
live system with a simulated environment created with a model. 

Customer Satisfaction. The benefit can be in terms of a reduction 
in time spent analyzing data and an ability to answer questions with more accuracy in a 
timelier manner. 

e. Quantify the Cost and Benefits of Alternatives/Capabilities 
Reprise 

Identifying alternatives can be the most difficult portion of the process. 
Benefits must be viewed primarily in terms of measurable value. Expected benefits 
should flow from the clear operating vision developed in Step 2 of the business case. 
Enigmatically, there are both quantifiable and non-quantifiable benefits. The former 
usually have some tangible or readily identified returns; the latter have less so. 
Additionally, there may be benefits that have no intrinsic value to one program but 
provide value to others which are called external benefits. 

(1) Quantifiable Benefits. These include cost savings, time 
improvement, acceleration of deliverables, quality enhancement and, in most cases, cost 
avoidance that is directly related to the program. The alternatives must also consider 
existing systems and programs. If improvements are to be measured from an “as-is 
baseline” - then it is not advisable to start from ground zero. It may be possible to look 
at related M&S domain initiatives in other programs, assess their applicability, and 
leverage them for success. The cost associated with using these existing alternatives 
should be less, given that a majority of the investment would be a sunk cost borne by the 
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previous developers. Similarly, it also advisable to consider partnering with another 
program and thereby sharing costs. This option is a possible alternative to reducing the 
up-front investment required for a new development effort. One should ensure that they 
have examined all potential benefits by using published references and experts in the 
field of cost analysis. 


(2) Non-Quantifiable Benefits. Traditionally, the issues of risk 
reduction, organizational efficiency, technology transference, product safety, and 
environmental impact reductions have been considered as non-measurable and therefore 
non-quantifiable. However, these are important issues, and one must consider them in 
their analysis. To illustrate this point, technology transference will be addressed. 

M&S technology transference can significantly influence costs, but 
in today’s DoD environment it has yet to receive adequate attention. Given the shrinking 
public purse and the demand for greater accountability and responsibility for the dispersal 
of funds, all must show due diligence in their public spending. They must consider the 
residual benefits of technology transference. 

Some M&S investment might be of use to other projects and Program 
Managers. For example, the Grizzly PM invested heavily in chassis M&S 
to support short-term design and performance analysis. This M&S 
investment resulted in $21 million worth of quantifiable benefits to the 
Grizzly program. The Grizzly PM funded the M&S effort through internal 
reallocation of funds. The PM’s supervisor, PM Combat Mobility 
Systems, recognized the potential for the use of these models for both 
other program requirements and in other programs sharing the Grizzly’s 
chassis. He leveraged the Grizzly Program’s M&S investment, securing 
funding to expand the applicability of the initial investment into other 
programs and to support other long-term Grizzly requirements. (Brown et 
al., 2000) 


When forecasting near-term savings in design and production 


costs, one company in the ICAF study accrued substantial non-quantifiable benefits. 
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“The engineers made a substantial leap in M&S knowledge when learning how to define 
data needs, how to shape models, and how to refine simulation runs, to narrow the 
bandwidth of problem solving. The resulting expertise, data and process could be applied 
to future projects.” (Brown et al., 2000) Not surprisingly, the company’s models and 
databases are the envy of the industry based on their visionary efforts. These efforts 
helped the company achieve a competitive advantage that was a non-quantifiable g ain 

Unfortunately, many decision-makers and PMs dismiss the concept 
and viability of non-quantifiable benefits. “Organizations rarely track non-quantifiable 
benefits or those outside the program’s realm with any real vigor. They don't afford them 
reasonable weight when analyzing costs and their alternatives.” (Brown et al., 2000) 
Similarly, external benefits almost always exist. Usually, these benefits that are 
applicable to the Government and DoD at large are not acknowledged due to the not 
invented here mentality. 

External Benefits are benefits which do not bring direct return or 
savings to the unique program being managed, but have applicability to other 
organizations. As mentioned previously, many M&S initiatives and their products can 
either be modified or directly transferred to other programs. Looking again to the Grizzly 
program, the contractor supporting the PM, United Defense Limited Partnership, 
developed a common product model database that benefited efforts at Aberdeen Proving 
Ground, The Army Warfighting Analysis and Integration Center, Waterways Experiment 
Station and National Training Center projects (Brown et al., 2000). This example shows 
that there is a residual savings or external benefits for follow-on users. PMs have 
identified problems with the high cost of collecting data and maintaining the databases 
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which can benefit other programs in the future. There is no focus in the acquisition 
community to address and incentivize PMs to address the external benefits of their 
programs. While the cost of performing this might be high, or perhaps even prohibitive 
to one project, it could be cost effective to several other end users. “Institutional bias 
forces PMs to ignore external benefits. A PM has no incentive to take on an M&S 
investment unless he can justify the expense from his existing program.” (Brown et al., 
2000) Even though, according to DODI 7041.3, societal costs and benefits outside the 
Federal Government are usually not included in a DoD Analysis, it is pertinent from a 
business perspective to consider external benefits. 

Perhaps the real value of identifying quantifiable and non- 
quantifiable benefits is in helping others outside the program to realize potential 
synergies of reuse. For example, the Program Executive Officer (PEO), who is charged 
with oversight of this program and many other related programs, will have better 
visibility into requirements and the potential benefits. He can more accurately assess 
M&S investment in relation to a broader spectrum of programs. Operational analysis and 
training are just a few examples of synergies of reuse between acquisition communities 
where M&S products developed for testing can be transferred and used for other uses. 
Many of these benefits, while external to an individual PM, may be internal benefit from 
the PEO’s perspective. The PEO must be provided with the data and information drawn 
from the business case analysis performed by an individual program under his purview in 
order to make sound management decisions. Based on the information provided in the 
business case analysis, the PEO can choose to redirect funding from other sources into 
the program, and/or direct a PM to take a course of action. This PEO action may not be 
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cost-effective in a micro-perspective for an individual program; but will bring an 
aggregate gain that far outweighs the individual investment. (Brown et al., 2000) 

4. Step 4. Evaluate Alternatives 

To evaluate the alternatives, the costs and benefits of each alternative must be 
compared and then ranked. Such comparisons must be accomplished using both 
quantitative and qualitative techniques and criteria. Quantitative techniques include net 
present value, benefit cost ratio, return on investment, payback method, internal rate of 
return, hurdle rate, and cost effectiveness analysis. Qualitative evaluation considerations 
such as relationship to business strategy, schedule risk, organizational and technical risks, 
social benefits, and legal and regulatory requirements may greatly alter the quantitative 
ranking. 

The choice of appropriate tools is program and situation dependent, and can 
greatly influence the outcome of the analysis. These tools will aid decision-makers in 
accurately evaluating all alternatives such that all costs and benefits are viewed on a level 
playing field. In general, each feasible alternative, life-cycle costs and benefits are 
adjusted using discount factors to account for the time value of money. A complete 
analysis properly relates quantitative and qualitative factors. Given the importance of 
these choices, one should seek expert advice and guidance before proceeding. 

One such tool is the Cost Analysis Strategy Tool (CASA). CASA is ideal for 
conducting life cycle cost estimates to include supportability-related trade-off analyses, 
sensitivity analyses and comparing different systems and alternative support structures. 

The CASA model is a life cycle cost decision support tool for Program Managers 
responsible for materiel acquisition systems; however, it can be reasonably used for the 
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acquisition of models and simulations. In particular, CASA addresses the Total 
Ownership Cost (TOC) for the objective system including research, development, test, 
and evaluation; manufacturing development and production; and the entire operational 
life during which the system must be supported. Virtually, every cost associated with a 
system is covered by CASA, whether one-time, recurring, or annual. 

The CASA model has extensive analytical capabilities. In addition to calculating 
life cycle cost estimates and identifying cost drivers, CASA also performs trade-off and 
sensitivity analyses. A wide range of sensitivity analyses can be conducted on the 
various cost parameters included within the CASA model. With this capability, the user 
can examine the cost impact of varying factors such as support equipment availability or 
the turnaround time for spare parts. The production rate and quantity buy analysis option 
assists users in determining the optimum quantity of times to procure. The robust CASA 
life cycle cost model can consider life-cycle studies for projects that last up to 50 years 
and accommodate customized maintenance schemes with up to 10 levels. CASA is a 
powerful tool for developing life cycle cost estimates and gaining a better understanding 
of the resultant cost figures through trade-off and sensitivity analyses. CASA 2000 is 
available online at http://www.logsa.army.mil/alc/casa. (McPherson, 2001) 

a. Quantitative and Non-Quantitative Evaluation Methods 
Simulation investment alternatives should be evaluated using multiple 
decision attributes that include both financial and non-financial criteria. The system or 
process for analyzing costs and benefits associated with and investment should include 
qualitative and quantitative criteria of a financial and non-financial nature. 
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(1) Quantitative Methods. This section addresses the 
quantitative methods of estimating and comparing costs and benefits of different 
alternatives. This section will refer often to entries developed in Figure 1. Benefit-Cost 
Calculations in Step 3 of the business case. Figure 1 provides a model and worksheet 
that can be used to perform the quantitative analysis and calculations of the benefit cost 
analysis. This section will provide the quantitative analysis methodologies. 

(2) Net Present Value (NPV). Per OMB Circular A-94, NPV, 
the discounted monetized value of expected net benefits, is the standard criterion for 
deciding whether a Government program can be justified on economic principles. Net 
present value is calculated by assigning monetary values to benefits and costs, 
discounting future benefits and costs using an appropriate discount rate, and subtracting 
the sum total of discounted costs from the sum total of discounted benefits. Discount 
rates and the discount factors to be used are provided and defined in OMB Circular A-94. 

Present value analysis is based on the principle that benefits 
accruing in the future are worth less than the same level of benefits that accrue now and 
costs that occur in the future are less burdensome than the costs that occur now. The 
formula for calculating NPV is: NPV = Present Value of benefits - Present Value (PV) of 
costs. 

The present value of a benefit or cost is calculated by multiplying 
the amount by a discount factor. The discount factor is equal to l/(l+i)n where i is the 
discount rate and n is the number of periods over which discounting takes place. If the 
NPV is positive, the financial return on the project is economically acceptable. This is 
because the cash flows generated by the investment over its economic life will: recover 
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the original outlay and any future outlays, earn the desired return on the outstanding 
balance, and provide a cushion of excess economic value. If the NPV is negative, the 
project is not acceptable on economic grounds. 


Additional methods of evaluating costs and benefits are available 
to help distinguish among alternatives with similar NPVs or ones where it is difficult to 
estimate present value. OMB Circular A-94 recommends that when a net present value 
cannot be calculated, agencies provide a comprehensive enumeration of the different 
types of benefits and costs and/or quantify benefits and costs even though it may not be 
possible to monetize them. 

(3) The Benefit Cost Ratio (BCR) or Profitability Ratio. The 
BCR measures the economic desirability of an investment by dividing the present value 
of its benefits (cash inflows) by the present value of the costs (outflows). The alternative 
with the highest BCR is the most cost effective because it returns the most benefits per 
dollar spent. The formula for this ratio is: BCR = PV (benefits)/ PV (costs). The BCR 
provides a measure of the benefits obtained per dollar spent. The higher the BCR the 
larger the return. Whereas the NPV is an absolute measure that refers to a specific set of 
values, the BCR allows comparison of different projects. In selecting among alternatives 
the BCR shows which alternative provides the largest return relative to costs. 

(4) Return on Investment (ROI). The ROI ratio is calculated 
by dividing the average annual operating cash inflow (benefit) by the annual net 
investment. All this ratio does is calculate what percentage of the investment the annual 
benefit cash flow is. This amount is calculated on an annual basis and the formula for 
this ratio may be calculated as: ROI = Average annual operating cash inflow / Net 
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investment. This ratio is limited in its usefulness because it does not take into 


consideration the economic life of the project and assumes a constant annual return. The 
ratio also ignores the time value of money and, therefore should not be used except in 
relation to annual returns or when speaking in generalized or gross terms. 

(5) Payback Method. This method estimates the time it takes 
to recover the original investment outlay. This value is calculated by dividing the net 
investment by the average annual operating cash inflow: Payback (time) = Net 
investment / Average annual operating cash inflow. This ratio gives a very rough test as 
to whether the investment will be recovered within its economic life span, however this 
ratio is limited in its use as it is insensitive to the economic life span and assumes 
constant annual operating cash inflows. It does not consider cash flows beyond payback 
and, therefore, does not measure profitability. 

(6) Internal Rate of Return (IRR). The IRR method determines 
the discount rate that makes the net present value of a project equal to zero. When 
applied to both cash inflows and cash outflows over the project’s economic life, it 
provides a zero net present value so that the present value of the inflows exactly equals 
the present value of the outflows. 

(7) Hurdle Rate. The hurdle rate is a minimum standard for the 
return required of an investment. A hurdle rate may be used to help select from among 
alternative investments when other decision criterion is lacking. For instance, a hurdle 
rate equal to the cost of capital as reported by the Treasury Department may be used. 

(8) Cost Effectiveness Analysis. OMB Circular A-94 states 
that cost-effectiveness analysis is appropriate wherever it is unnecessary or impractical to 
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consider the dollar value of the benefits provided by the alternatives under consideration. 
A program is cost effective if, on the basis of life cycle cost analysis of competing 
alternatives, it is determined to have the lowest costs expressed in present value terms for 
a given amount of benefits. Cost-effectiveness analysis is appropriate whenever: (1) each 
alternative has the same annual benefits expressed in monetary terms or (2) each 
alternative has the same annual effects, but dollar values cannot be assigned to their 
benefits. Cost-effectiveness analysis can also be used to compare programs with 
identical costs but differing benefits. In this case, the decision criterion is the discounted 
present value of benefits. 

b. Non-Quantitative Evaluation Considerations 

A non-quantitative evaluation approach often has to be used when there 
are limited measures or metrics available for a proposed simulation approach. 
Qualitative evaluation considerations including non-quantifiable or monetizable benefits 
may override quantitative criteria in the ranking or acceptance of projects. Such 
considerations include: relationship to business strategy, schedule risk, organizational and 
technical risks, social benefits, or legal/regulatory requirements. Non-quantifiable 
considerations for evaluating alternatives should be identified in the BCA. 

(1) Identifying and Evaluating Risks. Benefit and cost 

estimates are typically uncertain. Having a strategy to deal with the risk that is inherent 

in large simulation investments/projects is critical. One of the greatest risk factors to the 

success of simulation projects is the amount of development that is planned. Full-scale 

development is where the potential is greatest for significant cost and schedule overruns 

and lowered performance goals. The types of risks encountered in a simulation project 

may include: schedule risk, risk of technical obsolescence, cost risk, technical feasibility, 
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dependencies between a new project and other projects or systems, and the risk of 
creating a monopoly for future procurements. 

(2) Risk Management. Risk management is an organized 
method of identifying and measuring risk and developing, selecting, and managing 
options for handling risks. Risk management consists of four elements: risk assessment, 
risk analysis, risk treatment and risk management plan. Risk assessment which identifies 
and assesses all potential risk areas, any parts of a project where there is uncertainty 
regarding future events that could have detrimental effect on meeting the program goal. 
Risk assessment continues throughout the life of the project as previous uncertainties 
become known and new ones arise. Risk analysis characterizes each risk as to the 
likelihood of its occurrence and the severity of its impact. It results in a watch list of 
potential areas of risk. Risk analysis also continues throughout the life of the project. 
Risk treatment determination is made after risk has been assessed and analyzed. During 
risk treatment, a determination is made for how to deal with it. Alternatives include: 

• Transfer - risk may be transferred to another program or phase of the 
program, another Government service or a contractor. 

• Avoidance - it may be determined that the risks of any particular 
solution/altemative are too great and the alternative should be removed 
form further consideration. 

• Reduction - necessary measures can be identified to minimize the 
likelihood of a risk occurring and/or minimize the damage of its impact on 
program goals should it occur. 

• Assumption - a decision may be made to assume a risk if effective control 
can be exercised, the probability of risk is small, or the potential damage is 
either minimal or too great for another program or Government service to 
bear. 

• Sharing - if a risk cannot be appropriately transferred and should not be 
assumed, it can be shared with another program or Government service, or 
a contractor. 
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(3) Risk Management Plan. A risk management plan should be 
developed that includes information on the types, probability and impact of risks 
pertinent to the simulation project, including the risk that the funding request will not be 
approved or not approved in its entirety and plans for how to treat and manage the risk, to 
include how to respond to lower funding. Furthermore, requiring a higher return for 
projects determined to be of higher risk can accommodate risk. Also, risk analysis 
estimates of the probability that a simulation investment will fail and the impact this 
would have on the business can be subtracted from the expected benefits to adjust the 
ROI or NPV calculations to reflect risk. 

Sophisticated risk assessment methodologies, such as, probabilistic 
simulation can be used to estimate ranges for total annual cash flows and key variables 
can be identified. Probability distributions can then be assigned to the outcomes for each 
of the variables. Computers can be used to run multiple iterations. A contractor can 
perform an independent risk analysis of the selected approach or alternative and the ROI 
can be adjusted accordingly. 

5. Step 5. Conduct Sensitivity Analysis 

Sensitivity analysis is an essential step in the decision process as it accounts for 
ever-present uncertainties and variable changes. Such analysis repeats the evaluation of 
alternatives performed in Step 3 but with changes to the uncertain variables and examines 
the effect on the final decision. The outcome of the sensitivity analysis will provide a 
better understanding of the robustness of the output of the business case analysis. 

Sensitivity analysis is highly recommended, even if there appear to be significant 
differences among the alternatives, because an apparently superior solution may be very 
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sensitive to changes in a single variable. Sensitivity analysis is required when differences 
among alternatives are less obvious and may be totally driven by variability of key input 
factors. “The key factors to be tested may include, but are not limited to; project or 
program length, volume or quantity and mix of production units, requirements, 
configurations, assumptions, and discount rates and other economic factors.” (Brown et 
al., 2000) 

To conduct the sensitivity analysis, all parameters in the analysis are held constant 
except the factor being tested. The analysis is then reworked using different estimates for 
the factor under review. If this results in changes to the ranking of alternatives, the 
analysis is sensitive to that amount of change in the variable. Each parameter should be 
tested individually to determine its effect on the analysis. 

Sensitivity refers to the relative magnitude of change in one or more elements of 
an economic analysis that will cause a change in the ranking of alternatives. Sensitivity 
analysis is used for assessing the extent to which costs and benefits are sensitive to 
changes in key factors. In a sensitivity analysis, if one particular factor or cost element 
can be varied over a wide range without affecting the ranking of alternatives, the analysis 
is said to be insensitive to uncertainties regarding the particular event. A sensitivity 
analysis can provide a range of costs and benefits that are likely to be a better guide then 
a single estimate. 

If there is certainty and the preference ranking establishes one alternative as 
markedly superior to the rest, sensitivity analysis is probably unnecessary. However if 
there is uncertainty with at least some of the assumptions and the alternative of choice is 
not clearly preferable to the rest, then a sensitivity analysis may be necessary. 
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As part of the sensitivity analysis, major assumptions should be varied and net 
present value and other outcomes recomputed to determine how sensitive outcomes are to 
changes in the assumptions. Assumptions deserving the most attention will depend on 
the dominant benefit and cost elements and the areas of greatest uncertainty. For each 
alternative, key high risk factors should be changed to a less favorable number to test 
sensitivity. Key elements to evaluate include: length of project life; volume, mix, or 
pattern of workload; requirements; configuration; assumptions; discount rates; and cost 
and benefit estimates. (GSA, 2000) 

6. Step 6. Developing a Migration Strategy 

After determining the best alternative, one must develop a sound implementation 

plan to migrate the chosen strategy into the program (Brown et al., 2000). Cost, schedule 

and performance goals must be formalized in order to manage the investment/project. 

The plan for migration must incorporate a systematic approach to ensure the developer 

implements the identified drivers and captures the expected benefits. The performance 

goals should stem from the needs/requirements that alternatives are fulfilling and should 

address the benefits they are expected to provide. Performance goals should be 

consistent with the organization’s strategic plan goals and should be linked to or part of 

performance plan goals and measures. Schedule and cost goals must also be established 

to help ensure projects adhere to planned costs and schedules. Interim annual goals and 

measures must be established for multi-year projects to ensure timely detection of 

problems and implementation of corrective action. Implementation of the migration 

strategy will undoubtedly force changes to the program’s plan and budget. If a new tool 

or process is expected to save money, then those savings should be subtracted from that 

part of the program budget and reassigned elsewhere as an up-front action. 
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7. Step 7. Monitoring the Process and Assessing Results through 
Formalized Feedback 

The final step in developing a business case will be to create metrics to assess 
progress towards the overall vision. These metrics should be tied to the changes made to 
the program’s acquisition plan, to provide timely feedback on their success in meeting 
desired results in performance, schedule, and cost. Performance metrics should stem 
from the needs and requirements that alternatives are fulfilling, and should address the 
benefits they are expected to provide. Schedule and cost metrics must also be developed 
to help ensure programs adhere to planned costs and schedules (Kidwell, 1998; GSA, 
1998). 

Since PMs face tremendous pressure to bring a product into use, they must deliver 
their programs with complementary benefits first. This is their true priority, but they 
should also identify real or potential external benefits up the management chain to the 
PEO. That office can then make more informed decisions on the macro benefits. PMs 
should consider increasing investment earlier in the program if the business case strongly 
indicates downstream savings as a result. PEOs can provide the attendant oversight and 
direction, with a requisite re-allocation of funds when it is in the DoD’s best interest to do 
so. (Brown et al., 2000) 

Monitoring needs to be a truly integrated process, with all elements actively 
involved. A sharing of analysis, combined with a DoD commitment to maximize and 
optimize any potential benefits of M&S technology, will bring unprecedented reward in 
cheaper, better, stronger products and the associated prudence in managing the public 
resources. Conducting a Post Implementation Review that includes financial data, using 
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measures and metrics developed enables the monitoring; feedback and continuous 
improvement of the M&S product and development process. 

a. Conduct Post Implementation Review 

Once a system becomes fully operational or implemented, a Post 
Implementation Review (PIR) will be conducted. This review should occur about 3 to 6 
months after the project has become operational. It is highly recommended that the 
review be conducted by a group other than the IPT that has been responsible for the 
development of the system. This ensures that it is conducted independently and 
objectively. Subsequent PIRs should be conducted on a periodic basis after the first PIR 
to ensure that the completed system is continuing to meet organizational and user needs. 

Each PIR that is conducted has a dual focus. First, it provides an 
implementation assessment of the system, including an evaluation of the development 
process. Secondly, it indicates the extent to which the business case steps are sustaining 
or improving the success rate of M&S projects. The following areas should be evaluated 
as part of a complete PIR. Each topic should be documented with a summary of findings 
that support the conclusions and recommendations. Additional Documentation and 
Project Risk Checklists are included in Appendices D and E. 

b. Post Implementation Review Assessments 

(1) Mission. An analytical approach should be taken to 

determine whether the implemented system has achieved its proposed impact on the 

agency’s business. It is important that all agency M&S investments are aligned with the 

organization’s mission and the agency’s program objectives. Additionally, M&S 

investments are to be analyzed and evaluated in respect to the overall benefits for DoD 

business practices. The PIR team is obligated to determine the status of several project 
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variables, including the delivery of services or products estimation of cost savings, 
compliance with the DoD High Level Architecture, evaluations of the information 
product, e.g. accuracy, timeliness, adequacy, and appropriateness of information, 
identification of additional maintenance and security. 

(2) Systems Description. A specific description of the 
functions of the system should be documented. The tasks performed and the approach 
taken to accomplish each task and the resources used also needs to be outlined. All 
hardware, software, and applications software associated with the system should be 
documented as well. All personnel requirements and geographic locations that provide 
input, receive output, or assist in system processing should be identified. Finally, there 
should be an explanation of how the system contributes to the organization’s mission. 

(3) Change Control. The Change Control process and 
procedures for the system should be documented and evaluated for efficiency. A 
determination should be made of the number and severity of the changes to date and their 
impact on the stability of the system. An assessment should also be made describing the 
system’s ability to respond to changing requirements. 

(4) Operation. An analysis of the system operation, including 
hardware, system and application software should be conducted and compared against 
those projected. Finally, recommendations regarding system changes and redesign based 
on projected comparisons and operation problems. 

(5) Security. A security evaluation should be conducted to 
verify that the appropriate security requirements are documented and enforced. If 
problems are identified in this area, these should be outlined and corrective actions need 


59 





to be identified. Any security or risk incidents need to be identified and analyzed for 
potential system weaknesses. An evaluation should be made of the cost effectiveness of 
system security measures and recommendations made where improvements can be made. 
Finally, the contingency plans need to be checked to ensure that they are current and are 
feasible to minimize loss from threats and equipment/software malfunctions. 

(6) Outputs. The outputs of the new system, e.g. reports, data, 
or formats, need to be compared to those that were initially proposed. The impact of any 
changes on the initial design, geographic locations, or telecommunications factors should 
also be evaluated and documented. 

(7) Documentation. Any system documentation such as User’s 
Guides or Operations Manuals should be reviewed for completeness, accuracy, and 
timeliness. A list of all required documentation should be developed and kept up to date. 

(8) Management. A review of the support organization 
structure should be examined. The organizational structure and responsibilities as 
implemented should be compared against those documented during the project. The 
system ownership and individual authorities and responsibilities should be verified and 
updated, as required. Any areas where there is conflicting, unidentifiable or 
inappropriate management or supervision should be identified and corrected. Training 
issues should also be examined in this area to ensure that personnel (users and support) 
are properly trained. (GSA, 2000) 

c. Review and Evaluate Project Information 

In addition to the items reviewed in the Post Implementation Review, the 
following areas need to be evaluated and assessed: 
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(1) Technical Capability. The technical capabilities of the 
project, both current and future, should be reviewed and evaluated. Factors such as the 
competency of the workforce to use the new system and employee satisfaction or 
retention, the extent to which advanced technology was used, and the methodological 
expertise of the development team should be considered. 

(2) Measurements of Actual vs. Projected Performance. The 
project’s actual results should be compared to planned estimates in terms of cost, 
schedule, performance, and mission improvement outcomes. An attempt should also be 
made to determine the causes of major differences between the planned and final results. 

(3) Evaluation of Outstanding Issues. If the PIR reveals issues 
that still require attention, these issues need to be identified and documented. The issues 
should clearly document the estimates of cost and time, the risks for not addressing the 
issue, any tradeoffs or alternatives, and provide a recommendation from the PIR review 
team. The issues should then be sent to senior management for evaluation and a final 
decision on the actions to be taken. 

d. Review and Update Financial Information & Performance 

Measures 

Once the actual final financial and performance measurement information 
for the initiative has been gathered, it should be compared against the planned results. 
This will allow a determination to be made as to the success of the initiative and to 
determine the causes of any differences between planned and actual results. The 
following five areas should be evaluated in this process: 

• Evaluation of Cost Information - compares the actual versus planned life 
cycle costs for the initiative. 
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• Evaluation of Financial Return Information - compares the actual versus 
planned results for financial performance measures. 

• Evaluation of Non-Financial Return Information - compares the actual 
versus planned results for non-financial performance measures. 

• Evaluation of Acquisition and Procurement Information - compares the 
actual versus planned results for contract and contractor information. 

• Evaluation of Budget and Financing Information - compares the actual 
versus planned results for funding source and general budget and 
financing information. 

The monitoring and feedback step continues throughout the development 
and post deployment of the system. The collection of system, process and financial 
information creates a M&S development and deployment which can be audited and 
additionally, helps the PM achieve his programmatic goals. 
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IV. APPLYING THE BUSINESS CASE FRAMEWORK TO 
DEVELOPMENTAL TESTING 


A. INTRODUCTION 

This part of the thesis will take the business case developed in the previous 
section for programs in general and apply it to Test and Evaluation (T&E). The 
remainder of the thesis will deal primarily with the discipline of Developmental Testing 
(DT). T&E are essential parts of the development and deployment of all Army systems. 
“The information generated as a result of T&E influences every action taken during the 
system acquisition process.” (HQDA, 1997) The Developmental Test Command (DTC), 
a subordinate command of the Army Test and Evaluation Command (ATEC), has the 
roles of developmental testing and safety certification, and advises program managers as 
part of IPTs. 

DTC has a long history of using physical simulation as part of the process to 

confirm a developmental weapon system’s readiness and technical maturity. M&S 

within the T&E process is the integration of a mix of computer simulation, actual 

warfighting systems, and weapon system simulators, immersed in synthetic 

environments, distributed geographically, and connected through high-speed networks. 

Simulation for T&E includes Software-in-the-Loop (SWIL), Hardware-in-the-Loop 

(HWIL), or Human-in-the-Loop (HITL) simulations as well as synthetic environment 

simulations. T&E uses M&S to identify test parameters and drivers for field tests; 

determine high risk areas; predict test results; assist in the allocation of scarce test 

resources; and provide entity stimulation in support of interoperability testing. M&S 

provide the only route to obtain risk-reducing system-performance data in situations that 
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cannot be tested due to safety, cost, or other constraints. Presently, DTC is reengineering 
the Army’s technical test capabilities through its VPG initiative. The VPG consists of a 
comprehensive and interrelated set of synthetic environments, stimulators, and simulation 
test procedures operating within a standard architecture framework. Historical data from 
hardware tests on DTC ranges and other real-world test sites provide ground truth data 
that anchor the VPG to reality. 

DTC seeks a sound business methodology to provide cost avoidance and Return 
on Investment (ROI) data for future tests using M&S. DTC needs the data to ensure that 
it can recommend the right mix of live testing, which validates the M&S, and use of the 
VPG or virtual testing, that augments the live testing. The cost data help DTC encourage 
investment in its live and VPG facilities and technologies. “M&S is useful for aiding in 
test design and making pre-test predictions. Test results are invaluable in validating and 
improving models and simulations” (Coyle, O’Bryon & Hillegas, 2000). These data will 
be most credible to customers if developed as applications of Government approved 
methodologies. DTC therefore seeks to ensure that its VPG cost estimation 
methodologies are consistent with Government guidance on the role of T&E in system 
acquisition (HQDA, 1997), on cost estimation (CEAC, 1997), and on economic analysis 
(CEAC, 1995). This chapter will show how the business case for M&S, presented in 
Chapter 3, can accomplish this goal. The business case approach can help illuminate the 
point that “M&S can be more effective and efficient when the program office develops a 
formal plan early in the program and secures a ‘buy-in’ to the plan by the acquisition 
decision-makers and the testing community” (Coyle, O’Bryon & Hillegas, 2000). 
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DTC sponsored studies to assess the benefits of using virtual testing techniques in 
three historical programs. The PMs of the weapon system acquisition programs would 
ultimately realize these benefits. The historical programs were the Enhanced Position 
Location Reporting System (EPLRS) (Brugh, 1997), the Automatic Chemical Agent 
Detector Alarm (ACADA) (Brugh, 1997), and the Abrams M1A2 Tank (Brugh, 1996) 
technical test programs. Thefocus of the studies was to estimate the actual and potential 
savings to cost and schedule that were achieved through the use of virtual test techniques 
when compared to using live test techniques.. Another DTC study estimated cost 
avoidance at the Simulation/Test Acceptance Facility (STAF) (Johnson, 1996). It is a 
goal of this thesis to apply the business case methodology developed in Chapter 3 to 
these previous studies to validate the need for the use of such a business case in all major 
decisions involving the use of M&S in developmental testing. 

B. BUSINESS CASE APPLIED TO DT CASES 

1. Step 1. Baseline 

Since the key in determining the success of the selected programs lies in 
establishing their initial state before any changes in procedure were considered or 
implemented, the performance baseline measurements for the EPLRS, ACADA, M1A2 
and the STAF will be addressed herein. This will be done in order to show the 
changes/improvements the programs underwent during the different testing events. The 
baseline measures were taken using live testing techniques without the use of formal 
M&S techniques. 

The baseline objective was to compare the cost/schedule of the virtual testing with 

the same amount of testing using live techniques. Another major metric used for the 

studies was the “Number of areas assessed that are difficult/impossible to test physically 
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due to limitations in cost, time, manpower, or due to risk to humans, equipment, or the 
environment.” (Patenaude, 1996) This metric was used for all of the studied programs 
and was the main reason for the comparison between EPLRS I/II and EPLRS III. Cost 
and time would be prohibitive if human radio operators were used to obtain the required 
data. Other examples of this metric are Electromagnetic Interference, Nuclear 
Survivability, and environmental testing (Brugh, 1997). An additional metric was 
“Quality and quantity of test data.” (Patenaude, 1996) As might be expected, open air, 
live field testing with chemical agents is not practical/feasible. Chemical agent detector 
testing in the past has essentially always been conducted in environmentally controlled 
chambers; therefore, the past methodology for testing involving chemical agents was 
actually a “synthetic environment” in the broadest sense. The comparison was made 
between the cost/schedule during the ACADA Production Verification Test (PVT) using 
the Detector Test System (DTS) and the cost/schedule if the tests were conducted in a 
standard chamber. Since, the baseline should include a clear enunciation of assumptions 
and constraints, it should be understood that this is very difficult in the changing 
environment of M&S where techniques and technology opportunities are changing 
almost weekly. The main metric used was the “Number of areas assessed that are 
difficult/impossible to test physically due to limitations in cost, time, manpower, or due 
to risk to humans, equipment, or the environment.” (Patenaude, 1996) 

2. Step 2. Vision/Direction 

Current force reductions have lead to restructuring. DTC, in order to continue its 

testing mission effectively, must downsize and restructure to test smarter with fewer 

resources. Current and nascent technologies are the ideal means for accomplishing this 

goal. DTC has a long history of using physical simulation to confirm a weapon system’s 
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readiness and technical maturity; presently DTC is reengineering the Army's test 
capability through its VPG initiative. The VPG consists of a comprehensive and 
interrelated set of synthetic environments, stimulators, and simulation test procedures 
operating within a standard architectural framework. The VPG will be used to confirm a 
system's readiness and technical maturity, from concept through fielding, and with a 
substantial reduction in program resources. Historical data from actual hardware tests on 
DTC ranges and other real-world test sites provide ground truth data that anchor the VPG 
to reality. 

In order to prove the need to invest in new testing technology, DTC must prove 
the worth of developing a VPG on a continuous basis. The ultimate goals of VPG are to 
accomplish testing better, faster and cheaper. One method of proving the worth of VPG 
is through cost avoidance and benefit analyses. A cost avoidance methodology for VPG 
will be presented herein. Several studies have been conducted to show the results of 
some early VPG investments that benefited the testing community and the PM ultimately. 
The results of these studies will be used to demonstrate the validity of the cost avoidance 
model and the ROI. 

Most of the testers who discuss this subject agree that a VPG runs much the same 
as a physical proving ground, except that the nature of the item being tested, the 
mechanisms of communication, data collection methods, and other basic functions shift 
toward the digital computer model. DTC has tested physical hardware and prototypes on 
ranges and in laboratories for decades (physical testing); the VPG extends DTC’s test 
capability into the domain of models, virtual prototypes, and hardware-in-the-loop 
simulations (virtual testing). 
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Three testing paradigms, physical, and virtual, help one to distinguish among the 
paradigms and explain the realm of the virtual proving ground. Physical testing is a 
hardware process that yields original knowledge and experience pertaining to the 
functionality of the weapon system, subsystems, and components as well as the nature 
and effects of the natural and man-made environments (Ground Truth). Data from 
physical testing serves as the basis for functional and conceptual models of both the 
systems and the environments. Virtual testing is comprised of a mix of hardware 
systems, subsystems, components and/or prototypes; soldier-in-the-loop simulators; and 
digital models. Data collected from virtual testing is based on the system, subsystem, or 
component performance of the physical hardware or human in the loop simulations 
coupled with the modeled effects of the remainder of the system and environment. This 
virtual test data contains a level of uncertainty due to the assumptions and 
approximations used in the digital models. However, the application of a rigorous 
verification, validation and accreditation process can minimize this uncertainty. 

Given the above basic understanding of the realm of the VPG, it is reasonable to 
ask how the VPG operates and who the players are. The current VPG operational players 
include DTC test centers, Operational Test Command (OTC), Research and Development 
Centers (RDECs), PMs, and contractors. After full VPG development, the players will 
have additional testing tools in the constructive simulation arena that will allow testing 
that is no longer limited to physical proving grounds or test centers. For some completely 
constructive testing, the concept of taking a piece of hardware to a specific physical 
location will no longer be relevant. DTC will still be the tester in the sense that DTC test 
centers will produce models of the test process, and ground truth data needed for both 
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producing and validating the system and environment models. This role of the virtual 
tester is complementary to the classic physical testing mission that is necessary to 
continue regardless of the sophistication of the emerging VPG. Important players are the 
other members of the IPT and the PMs. It is reasonable to visualize the PM office for a 
developmental system as the organization that links to various players to obtain the 
necessary testing models and other pertinent information to run a test on a system model 
developed at the PM’s organization. For instance, running such a test involves s haring 
the system model and its associated test issues and criteria with the DTC agency, which 
tailors the model for the specific test. All of this takes place within a distributed, but 
connected, complex of organizations that contribute to the virtual testing process. 

As mentioned above, a reasonable place to start an analysis of the VPG process is 
the hardware testing process. Figure 2 is a simplified diagram of the process of 
conducting hardware tests at fixed ranges and test centers. Basically, the process 
involves a physical system that is designed and manufactured to satisfy a set of critical 
issues and associated criteria, and tested to those issues and criteria. A set of test plans is 

derived from the requirements of those issues and criteria. These test plans contain the 

* 

test procedures and call for specific instrumentation to obtain test data. Test resources 
and historical data also contribute to test plans. Tests are executed from the plans. Test 
outputs are in the form of test data. System assessors use the data to determine how well 
test issues are satisfied and criteria are met. Classically, DTC is the major player in 
essentially all of the boxes of this physical hardware technical testing process. This is 
due to the fact that the developer sends the hardware system to a DTC test center or 
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proving ground to be tested. Additional weapon system developer resources or 
equipment are often added at the test site or range to execute tests and obtain data. 


HARDWARE TESTING PROCESS 



Figure 2. Hardware Testing Process. (Source: Researcher) 

The virtual testing process. Figure 3, is diagrammed in much the same way as 
Figure 2, but differs in two basic ways. First, systems being tested and test procedures 
themselves tend to be in digital forms rather, than physical forms. The digital form 
includes the virtual testing zone where humans and hardware in the loop simulations are 
challenged by synthetic stimuli. Second, Figure 3 follows the VPG operational concept 
that distributes the responsibility for testing to those organizations that are a part of the 
distributed VPG complex. The testing process is on the centerline of Figure 3. 
Support/infrastructure are shown above and below the centerline. The Intemal/Extemal 
Interface Infrastructure represents principally the networked electronic infrastructure that 
links players in the VPG testing process. 
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Figure 3. Virtual Testing Process. (Source: Researcher) 


3. Step 3. Quantify the Costs and Benefits of Alternatives/Capabilities 

According to the Army, the “Economic Analysis Process” (CEAC, 1995) is an 
eight step process: (1) Establish objective. (2) Formulate assumptions, (3) Identify 
constraints, (4) Identify alternatives. (5) Estimate costs and benefits for each alternative, 
(6) Compare alternatives, (7) Perform sensitivity analysis, and (8) Report results and 
recommendation. The Department of the Army Economic Analysis Manual recommends 
that an economic analysis study plan include the mission, background, purpose, 
constraints, assumptions, cost element structure, cost and benefit estimating 
methodology, system description, configuration, schedules, and issues. The level of 
economic analysis detail increases with project dollar value or project visibility. Among 
the limitations with economic analysis discussed in the manual are that economic analysis 


cannot be applied with cookbook precision, but must be tailored to fit the problem, and 
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that it is not a substitute for sound judgment, management, or control. The Business Case 
Framework outlined in detail in Chapter 3 of this thesis has all of the steps outlined by 
the Army and features the characteristics necessary to make a business decision to make a 
decision about using M&S in the discipline of testing. Therefore, the techniques for 
economic analysis detailed in the Business Case Framework outlined in Chapter 3 of this 
thesis will be used to do the analysis of the four systems. 

a. Cost Analysis Metrics 

“A true measurable metric would be any numerical value that enables us 
to assess how much faster, how much better, and how much cheaper a given acquisition 
process is or can be fielded.” (Pinker, 1997) Measurable T&E cost metrics, live or VPG, 
include cost of consumable items, cost of facilities, cost of technology, and test-personnel 
costs. Others are schedule with consideration for acquisition phase time, administrative 
lead-time, and of course test time. Developmental T&E affect program cost through 
knowledge of performance and the design changes necessary to implement a system that 
does not meet requirements. Developmental T&E provide benefits in troubleshooting 
and data requirements generation. 

These and other metrics may apply to cost savings, cost avoidance and 
productivity improvements. Cost savings result in the reduction of cost of approved 
budget items. A cost avoidance is the result of an action taken in the immediate time 
frame that will decrease cost in the future. For example, investment in a test technology 
that decreases the test cost while increasing risk reduction is a cost avoidance action. A 
productivity improvement is a reduction in future personnel time and effort requirements 
associated with a function or assigned task that has been included in an approved 
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program. Under normal circumstances, productivity improvements do not represent an 
opportunity to reduce an approved program, budget or force structure. Unlike cost 
avoidance, productivity improvements have no direct impact on future requirements for 
funding, but enable accomplishment of more work with existing personnel. Productivity 
improvements can accrue at any time during the life cycle. 

b. Cost Estimation Approaches 

With a description of the process, contents and metrics for cost estimation, 
the analyst must select an approach. “The engineering approach, parametric approach, 
analogy approach, and expert opinion approach are four cost-estimating methods. The 
use of a specific approach varies with the reliability and quantity of available data. Each 
approach has limitations.” (CEAC, 1997) 

The engineering (bottom-up) approach is an examination of separate work 
segments in detail and a synthesis of the many detailed estimates into a total. With this 
approach, the analyst divides the system, activity, or item of hardware into its segments 
and makes an estimate of each segment's costs. The analyst then combines these 
estimated costs with estimates of integration costs to arrive at a total cost. A major 
limitation of the engineering approach is that it requires the analyst to have an extensive 
knowledge of the system, activity, or item. Also, the analyst must know both the 
development and production processes. Particularly for new technologies, the detailed 
knowledge required for a complete engineering analysis is not always available, m aking 
this approach the most difficult to apply. (CEAC, 1997) Since the technologies are 
known for the EPLRS, AC AD A, and Abrams M1A2 historical studies the engineering 
approach was used in the analysis. 


73 







The benefit estimating process is similar to that for cost estimating. Data 
must be collected from appropriate sources and analyzed; relationships among data must 
be identified. Inflation and discounting must be applied to annual dollar values via 
standard methods; the economic life of the alternatives and the fiscal years when benefits 
accrue must be carefully considered. Once benefits have been calculated, savings must 
be separated from cost avoidance and productivity improvements. 

c. VPG Considerations 

In the VPG cost estimating methodology, DTC wishes to separate 
operating costs from fixed or investment costs. The total cost of the current alternative = 
Fixed Costs( F a ) + Variable Costs (V a )- DTC wishes to make cost estimates for the PMs. 
The fixed cost to the PM may be zero if an acquisition program can use a maintained 
facility developed by a predecessor program, or new capability is developed by DTC. 
For virtual testing, the fixed cost is usually an investment in the model, simulator or 
stimulator. The Investment Cost, a nonrecurring cost, includes land, building, machinery 
and equipment. Operating Cost, or recurring cost includes personnel, maintenance, and 
materiel/supplies. F a includes instrumentation development cost, and facility 
development cost (materials and labor). V a includes labor, expendables, and maintenance 
and test preparation/planning. 

Cost estimates for the VPG are complicated due to the cost estimation of 
software development. “Because software life cycle costs account for a significant 
portion of information systems' costs, and are often significant in materiel systems, they 
must be estimated carefully. Software cost estimating involves a large degree of 
professional judgment, from both a project management and cost analysis perspective.” 
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(CEAC, 1997) To the extent that each program requires a degree of VPG ongoing 
software and technology development, the cost estimating process requires care. 

Since one of the motivations behind doing virtual testing is accomplishing 
testing faster and cheaper, a cost avoidance methodology would help define the benefits 
associated with this new testing paradigm. Based on the previous sections that help 
define what physical and virtual testing involve, a cost avoidance methodology can be 
developed. In order to develop a VPG cost avoidance methodology, the basic steps for 
economic analysis must be considered. The Seven Steps for Economic Analysis include: 
establish and define goals and objectives, formulate assumptions and identify constraints, 
identify alternatives for meeting objectives, determine inputs and outputs of each 
alternative, compare cost and benefits of alternatives, evaluate and determine the risk and 
uncertainties, prepare conclusion and recommendation. For this effort the focus will be 
to compare the cost and benefits of alternatives. 

When performing a benefit analysis, all significant benefits must be 
included whether quantifiable or non-quantifiable. Benefits, which cannot be quantified, 
should be described in narrative form. Every effort should be made to quantify benefits 
to the maximum extent possible. A sample of some benefits to PMs, testers and 
evaluators using M&S include: performing the mission with fewer test items, increased 
understanding of simulation technology, supporting safety and environmental 
assessments, preventing false starts by performing test rehearsals, better understanding of 
how the components/subsystems/systems under test are designed to function in an 
integrated fashion, more confidence in data due to verification by simulation, and 
providing a test diagnostic capability. Some of the benefits of using M&S to the 
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customer or Program Manager include: fewer test items, lower test costs, reduction in test 
time, early knowledge of test environment models, powerful tool for data analysis and 
trouble shooting, improved data requirements generation, and validation of 
component/system models. Once the benefits have been identified, then the benefits are 
subdivided into those that are dollar quantifiable and those that are quantifiable in other 
terms. 

The benefit estimating process is similar to that for cost estimating. Data 
must be collected from appropriate sources and analyzed; relationships among data must 
be identified; inflation and discounting must be applied to annual dollar values via 
standard methods; the economic life of the alternatives and the fiscal years when benefits 
accrue must be carefully considered. Once benefits have been calculated, savings must 
be separated from cost avoidances and productivity improvements. 

The three categories of quantifiable benefits include cost savings, cost 
avoidance and productivity improvements. A cost savings results in the reduction of an 
approved program. The item under consideration must be part of an approved budget. A 
cost avoidance is the result of an action taken in the immediate time frame that will 
decrease cost in the future. For example, an engineering improvement that increases the 
mean time between failures and thereby decreases operation and maintenance costs is a 
cost avoidance action. A productivity improvement is a reduction in future personnel 
time and effort requirements associated with a function or assigned task that has been 
included in an approved program. Under normal circumstances, productivity 
improvements do not represent an opportunity to reduce an approved program, budget or 
force structure. Unlike cost avoidances, productivity improvements have no direct 
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impact on future requirements for funding, but enable accomplishment of more work 
with existing personnel. Productivity improvements can accrue at any time during the 
life cycle. Following is a cost avoidance methodology developed to support the cost 
avoidance activities of the VPG for test articles as well as test resources. 

The developed VPG cost avoidance methodology uses a comparison 
analysis between the cost of the current alternative and the cost of the simulation 
alternative. This is similar to the business savings calculation in Figure 1 (Benefit Cost 
Calculations). 


Total Cost of Current Alternative (physical testing) = Fixed Costs( F a ) + 
Variable Costs(V a ); where the fixed cost may be zero. Total Cost for Simulation 
Alternative (virtual testing) = Fixed Costs (F b )+ Variable Costs (V b ); where the fixed cost 
is usually an investment. Cost Avoidance = Total Cost of Current Alternative - Total 
Cost for Simulation Alternative. 


Where: 

• Investment Cost (Nonrecurring cost) - Land, building, machinery, 
equipment and software. 

• Operating Cost (recurring cost) - Personnel, maintenance/updates 
(hardware and software), and materiel/supplies. 

• F a - Instrumentation Development Cost, Facility development cost 
(materials and labor). 

• V a - Labor, expendables, maintenance and test preparation/planning. 

• Fb - Simulation development/modification, instrumentation development, 
HWIL simulation facility development. 

• Vb - Labor for test execution, expendables, hardware and 
software/simulation maintenance, test preparation/planning. 

If the overall objective is to field faster, better, and cheaper weapon 

systems, then a true measurable metric would be any numerical value that enables us to 
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assess how much faster, how much better, and how much cheaper a given acquisition 
process is or can be. Some quantifiable measurable metrics are noted herein. 

• Cost with consideration for the consumable item price index and military 
specification conversion price benefit. 

• Schedule with consideration for the acquisition phase time, administrative 
lead time and test time. 

• Program cost with consideration for change in program cost as a 
consequence of changed acquisition processes. 

• Unit Life-Cycle Cost considering change in projected unit life-cycle cost 
as a consequence of changed acquisition process. 

• Operational performance versus cost with consideration for the 
comparison of operational test results versus specified performance. 

• Cost of Performance considering the kind of system performance that can 
be bought for a given cost. To derive this metric it would be necessary in 
some way to quantify various combinations of system performance. 

• Cost as an independent variable is savings in a program where costs are 
held constant and performance and schedule are adjusted. 

The primary metrics used in the following studies considered were cost 
and schedule even though others could be considered for future analysis. 

4. Step 4. Evaluate Alternatives 

a. Analysis of Historical Programs 

Where possible, the live test equivalent to a virtual test procedure was 
chosen or designed to provide the same level of risk reduction. It is not assumed that test 
money is limitless. Thus, if the live test design produced an unreasonable test cost, a 
reduced-scope live test was used for comparison, with an attendant increased technical 
risk. Labor costs are normalized to the same year, and not broken down by skill levels. 
There is a factor for inflation. The cost of facilities may be paid with DTC institutional 
funding. 
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b. 


Enhanced Position Location Reporting System Study 


The EPLRS Technical Test III (TT III) was conducted at the US Army 
Electronic Proving Ground (EPG), Fort Huachuca, AZ. A series of EPLRS tests (TT I 
and TT II) were conducted five years prior to TT III, using live testing (i.e., human radio 
operators/data collectors). The methodology used was to calculate the additional 
resources required, for the same number of test days (75) during TT III, using TT II 
(labor intensive) methods. 

The TT I/II test consisted of an average of 160 radio sets (RSs) distributed 
at various locations in an area of 400 km and required over 300 operators/data collectors. 
There was limited automated data collection. Test-scenario-control was by hand-held 
FM radio, and required extensive logistics support for the operators. All of the 
equipment at each site was installed and removed daily for security purposes, which 
lengthened the test day by about four hours. Data consolidation, reduction, and analysis 
normally took about five days. 

During TT III many time/cost-saving automated tools and methods for 
control, monitoring, and data collection were used. Elements contributing to cost 
avoidance were: less logistics support; number of operators; a Test Control Center 
(TCC), use of a Personal Computer (PC) as a Test Item Stimulator (TIS); and 
security/test sheds with alarms to house the equipment. The TCC, with automated data 
collection, reduced analysis time from five days to one. Each of the average of 120 sheds 
held a PC/TIS, the EPLRS, a packet radio for communications between the TIS and the 
TCC, and the alarm. In effect, the self-contained sheds with the TIS were a "virtual 
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workforce" that replaced the human operators/data collectors. Shed support consisted of 
exchanging batteries, turning on/off equipment, TIS data dump, and on-call maintenance. 


The original study (Brugh, 1997 A) assumed that the TCC, sheds, and 
automated data reduction tools were not available. Daily setup and tear down increased 
the workday by two hours and required large trailers for equipment storage. Support 
personnel increased by 50% from 69, plus 120 operators. Transport vehicles carried six 
testers and test equipment. One instructor was required per group of 16 to provide three 
days of training to military and civilian operators. The analysis constrained the test cost 
at $10M maximum, and assumed a civilian/military mix of testers to be 115/40. Civilian 
labor was $32/hr and military labor was free to the customer. The analysis did not 
consider the test center’s investment in PCs, packet radios, automated data collection 
software, and the cost of the TCC. 

The additional resource cost for 75 test days was $2.2M labor + $0.8M 
overtime + $0.1 M equipment, and fixed costs of $0.1 M training + $.03 M equipment for 
a total $3.2M. Since the TT III cost $6.3M as conducted, the $10M spending limit would 
be nearly reached ($3.2M + $6.3M) during the 75 test days of labor intensive testing. 
Expert opinion was (Brugh, 1997) that only one-third to one-half the amount of usable, 
high-quality data would be acquired during the 75 test days with this method. Another 
alternative to the earlier study allows use of the sheds, which would avoid the overtime 
costs. This still results in a $2.4M cost avoidance for the VPG stimulator technology. 

Using the VPG cost avoidance methodology presented previously, the 

following summaries can be made. The cost of the EPLRS TT I/II is $10.085M. The 

cost of the EPLRS TT III is $7.233M. Both test cost amounts have been adjusted to 1993 
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base year dollars. These figures were obtained by totaling the fixed costs of facilities and 
instrumentation and variable costs of vehicles, hand held radios and labor. Subtracting 
the cost of the virtual test from the physical test, the cost avoidance is S2.852M. The 
following savings, which used an estimated test costing analysis, were stated in the study: 
“EPLRS Savings as tested: Virtual testing during TT III, saved the EPLRS program $3.2 
million, provided many millions of dollars worth of risk mitigating and risk management 
information, and shortened testing by a minimum of 75 days.” (Brugh, 1996) Using the 
VPG cost avoidance methodology produces numbers that are marginally close to those 
stated in the study. See the test methodology comparison in Table 3. 



Human in the Loop 

Test Item Stimulator 


Required Personnel 

155 

69 


Test Days Avoided 

0 

75 


Cost To Test $ 

S10.085M 

S7.233M 

Cost Avoidance S2.852M 

Table 3. 

EPLRS Test Methodo: 

logy Comparison. (Source: Researcher) 


In summary, virtual testing during TT III resulted in substantial cost 
avoidance, $2.4M - $3.2M, and significantly increased quality and quantity of data, vice 
use of less VPG-intensive technologies. 

c. Advanced Chemical Agent Detector Alarm Study 

The chemical/biological defense case study was on the ACADA PVT at 
Dugway Proving Ground, UT. As might be expected, open air, live field-testing with 
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chemical agents is neither practical nor feasible. Chemical agent detector testing in the 
past has essentially always been conducted in environmentally controlled chambers, or 
using simulants in lieu of live agents (i.e., virtual testing). Previous testing was 
conducted in Building 3445, the large chamber facility. While the large chamber can 
accommodate testing of large vehicles and equipment, testing of small agent detectors 
required construction of a smaller chamber inside the large facility. Past testing indicated 
that an earlier small chamber design could simultaneously test only two detectors of the 
six for which it was designed. The number of support personnel for Building 3445 is 
fourteen. For the ACADA PVT, a new stand-alone, HWIL test facility (the Detector Test 
System, DTS) capable of testing twelve detectors simultaneously, and requiring only five 
testers, was developed. The customer contributed to the fixed-cost of construction of the 
DTS ($82,200 of a total cost of $421,000). Twelve detectors were tested simultaneously 
for 39 days in Phase I, eight for 92 days during Phase II. 

The study objective (Brugh, 1997) was to estimate cost for Phase I Agent 
Testing and Phase II Agent/Simulant testing during the ACADA PVT, using the DTS and 
using Building 3445. Testing using a corrected six detector chamber design was not 
considered. 

The study assumptions (Brugh, 1997) were as follows. Each tester costs 
$355 per ten-hour day. The times required to startup, conduct and shut down a trial are 
the same in both facilities. The PM had no fixed facility-usage cost and no cost for 
construction of a small chamber in Building 3445. Test equipment reliabilities are the 
same in both cases. 
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The operating cost is the number of people who run the facility multiplied 
by the personnel cost per day, multiplied by the number of test days. For phase I, the cost 
is S.065M. The corresponding Building 3445 cost would be greater by factors of 14/5 
(personnel) and 12/2 (detectors) to total S1.1M. For Phase II, the 92-day cost was 
S.154M. The corresponding Building 3445 cost would be greater by factors of 14/5 
(personnel) and 8/2 (detectors) or SUM. 

The variable cost totals are (Cost Phase I + Cost Phase II), S.219M for the 
DTS and S2.8M for Building 3445. The total cost to the PM, fixed plus variable, was 
S.302M for the DTS. The test schedule would increase from 131 days for the DTS to, in 
Building 3445, 602 days. For this study, we add an alternative scenario that assumes 
correction of the earlier design to allow simultaneous measurement of six detectors in 
Building 3445. This results in S.364M variable Phase I costs and S.575M in Phase II, or 
S.939M total cost. Schedule would be 201 test days. Investment in the DTS resulted in 
substantial cost avoidance to the customer even for this technically unsubstantiated 
scenario. 

The objective of the AC ADA study was to compare the cost and schedule 
of the virtual testing during the AC AD A Production Verification Testing, with the same 
amount of testing using previous testing techniques. As one would expect, live open air 
testing with chemical agents is not practical or feasible. Chemical agent detector testing 
in the past has essentially always been conducted in environmentally controlled 
chambers. This technique is essentially virtual testing. The comparison was made 
between the cost and schedule during the ACADA production verification testing using 
the DTS and the cost and schedule if the same tests had been conducted using the 
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Chemical Agent Monitor test configuration. The DTS, which uses more automated 
simulation techniques, will be considered the virtual testing method. The testing 
completed using Chemical Agent Monitor test configuration will be considered the 
physical testing technique. The factors considered when conducting a cost and schedule 
analysis of the two testing techniques include: number of personnel required to conduct 
testing, number of items which can be tested simultaneously, time required for test setup, 
and facility costs (procurement, setup, and sustaining). 

Again, using the cost avoidance methodology presented previously, the 
following summaries can be made. The cost of the ACADA test using the Detector Test 
System is $.302M. This figure was calculated using the fixed cost of developing the 
Detector Test System and the variable cost of labor. The cost of the ACADA test, if 
conducted using the Chemical Agent Monitor test configuration, would be $2.823M. 
This amount was calculated using the variable cost of labor for running a test using the 
Chemical Agent Monitor test configuration. Subtracting the cost of the virtual test from 
the physical test, the cost avoidance is $2.521M. The following savings, which used an 
estimated test costing analysis, were stated in the study: “ACADA Savings as Tested: 
Virtual testing during Production Verification Testing, saved the ACADA program $2.5 
Million and shortened testing by 471 days.” (Brugh, 1997). The numbers for cost savings 
stated in the ACADA study matched the numbers for the calculations using the VPG cost 
avoidance methodology. The $2.5 Million cost savings stated in the study are actually 
cost avoidance dollars by definition. See the test methodology comparison in Table 4. 
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Chemical Agent Monitor Test 

Detector Test Set 


Detectors Tested 

2 

12 

Test Days 

602 

131 

Cost To Test $ 

S2.823M 

S.302M 

Cost Avoidance S2.521M 

Table 4. ACADA Test Methodology Comparison. (Source: ] 

Researcher) 


The results were substantial cost avoidance, $2.6M, and schedule 
avoidance, 471 test days, from investment in the newer simulation technology vice usage 
the existing old technology. 

d. Abrams M1A2 Tank Study 

This study (Brugh, 1996) identified use of virtual testing during the 
Production Qualification Test (PQT) program and during the live fire testing at Aberdeen 
Test Center (ATC). Seven PQT tests at ATC and two at the White Sands Missile Range 
(WSMR) used virtual testing that resulted in cost avoidance. The WSMR 
Electromagnetic Environmental Effects test and Nuclear Effects test had no feasible live 
test alternative and therefore did not involve cost avoidance, but did result in risk 
reduction. For this work, only the accuracy test will be addressed. 

Within the accuracy test description, the first letter indicates tank motion 
and the second indicates target motion to be stationary (S) or moving (M). S/M, M/M, 
and M/S at Aberdeen Test Center’s Trench Warfare I (TW I), with targets simulated by 
laser, each costs 15 testers @ $400/tester/day (10 hr) for 22 days, or $.132M. Non-Firing 
Tracking M/M (NFT M/M) costs 10 testers @ $400/tester/day for 8 days or $.032M. The 
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total labor cost on TW I is $.428M. Expert opinion indicates that physical target 
simulators require at least twice as many days to collect the required data and two 
additional testers. A daily target cost of $500/target/day would accrue for the S/M, M/M, 
M/M NFT scenarios. For the S/M and M/M tests, the cost would be $.321M. The M/S 
cost is the same (S.132M). The M/M NFT cost would be $.085M. The total cost, labor 
and targets of the physical simulation tests are S.859M. The difference (S/M, M/M, M/S 
and M/M NFT) of labor and targets is $.431M. See the test methodology comparison in 
Table 5. 



Table 5. Abrams Ml A2 Test Methodology Comparison. (Source: Researcher) 


For S/S at Trench Warfare II, there was a requirement for measuring 
accuracy at four distances. Prior to TW II construction, firing projectiles at each of the 
required distances met the requirement. At TW II, virtual video “scoring” at the four 
distances, for a projectile fired at the longest distance, reduced by 75% the number of 
projectiles and the test time. This avoided almost $3M in the cost of ammunition alone, 
with added labor reduction of nearly $1M. The total cost avoidance by using virtual 
testing during the accuracy testing was $4.4M. Other virtual fire control tests brought the 
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total cost avoidance to $4.8M, a schedule reduction of 258 days, and 2580 projectiles not 
fired. 

e. Simulation/Test Acceptance Facility (STAF) 

The STAF is a HWIL facility at the Redstone Technical Test Center 
(RTTC) in Huntsville, AL that greatly reduces the number of destructive live flights of 
missiles in production flight-testing and in stockpile testing. Completed missiles 
containing tactical seekers, guidance electronics, inertial navigation systems, warheads, 
squibs, motor, and control actuators are tested in a remotely controlled bunker. With the 
motor and warhead circuits disconnected, the missile flight dynamics are simulated using 
a six degree of freedom digital model of the missile’s airframe running in real time. The 
facility can modulate radio frequency signals to present realistic in-band representations 
of complex targets to the millimeter wave seeker or realistic InfraRed (IR) signals to an 
IR seeker. A real-time data collection system records data from simulated launch to 
simulated target impact. Missiles are consumed only to test motors and warheads, and as 
model validation. Rounds tested in the facility can be returned to the inventory. Separate 
cost estimates (Johnson, 1996), not repeated for this work, indicate that the STAF with 
RF representation cost $6M. Typical live firing lot acceptance testing costs customers 
approximately $9.5M/year. The same testing conducted in the STAF costs 
approximately $1.5M/year for cost savings of $8M/year. 
f VPG Cost Avoidance 

The chart shown below summarizes the VPG cost avoidance to the customer 
through use of M&S to support the four acquisition programs addressed in this thesis. . 
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VPG COST AVOIDANCE (SM) 



EPLRSFY93 ACADAFY93 ABRAMS FY94 


STAFFY95 


Figure 4. VPG Cost Avoidance. (Source: Researcher) 


5. Step 5. Conduct Sensitivity Analysis 

The STAF facility was the only program that performed a detailed sensitivity 
analysis. The Air to Ground Missile System (AGMS) Project Office and DTC RTTC 
performed an analysis of the perspective factors and trade-offs involved in implementing 
a HWIL lot acceptance program versus a traditional Fly-to-Buy (FTB) program. This 
analysis considered the two alternatives available to Longbow Hellfire (LBHF), as well 
as, the historical data provided by the earlier HELLFIRE I. The primary factors in the 
development of the assumptions involved in this analysis process were as follows: 

• Unit Costs: 

• HELLFIRE cost per missile S 20k. 

• LBHF cost per missile $ 300k @ Low Rate Initial 

Production (LRIP). 

• Test Programs: 
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• HELLFIRE and LBHF Lot Acceptance Testing: FTB @4-10 
missiles per month. 

• LBHF HWIL Simulation: 4 live fires per year and up to 20 
missiles simulated flight testing per month. 

• Facilities: Facility for the HWIL Simulation: $ 5.8 million. The building 
and instrumentation. 

• Labor: Personnel needed to run this facility for one year: estimated at 
approximately $0.8 million. 

With these factors in mind, a trade-off analysis was performed using costs for four 
missiles with the results portrayed in Table 2. The total savings projected under a HWIL 
simulation lot acceptance test program would represent cost savings of $7.78 million per 
year over a conventional FTB program. Even including the $5.8 million necessary for 
developing the HWIL simulation facility, the payback period under this assumption 
would be less than one year. 


LONGBOW 

Lot Acceptance Methodology 
Cost Tradeoff 

(Post Cost Reduction Program) 


BASIC HF 
FTB 

MISSILE COST $1.15 MIL 
EGLIN SUPPORT $0.72 MIL 

RTTC SUPPORT $0.53 MIL 
SYS. SIM. SUPPORT $0 


LB HF 
FTB 

(What if) 

$8.11 MIL 
$0.72 MIL 

$0.53 MIL 
$0 


SIMULATED 

PROGRAM 

$0.68 MIL 
$0.06 MIL 

$0.44 MIL 
$0.40 MIL 


TOTAL 


$2.40 MIL 


$9.36 MIL 


$1.58 MIL 


SAVINGS = $9.36 MIL 
$1.58 MIL 


NONRECURRING 

Total Facility Cost: 


$7.78 MIL PER YEAR 
PAYBACK PERIOD « 1 YEAR 


FACILITY COSTS: 
PM-ITTS $1,000K 

other S330K 

AGMS-PMO S4.500K 
S5.830K 


Table 2. Longbow Lot Acceptance Methodology Cost Tradeoff. (Johnson, 1996) 
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The Basic Hellfire (HF) FTB column indicates approximately what AGMS pays 
per year for the HELLFIRE I FTB program, and also indicates where the true cost growth 
in FTB actually lies, namely, missile cost. This total amount includes missile cost, range 
support, and pre-flight support. The LBHF FTB column indicates what the cost would be 
if the AGMS Office implemented a similar FTB program with just four missiles per 
month. The tremendous yearly cost avoidance is due to the non-destructive nature of the 
simulated flights, which allows the rounds to be placed in inventory upon test 
completion. The analysis indicates a facility cost payback period of much less than one 
year. 

To determine an annual return on investment for the STAF facility, the ROI 
formula in Chapter 3 of this thesis was used. ROI = cash inflow (Benefit) / net 
investment. To account for the test and simulation environment the ROI formula was 
rephrased as follows: ROI = test/simulation workload / fixed cost of facility and 
instrumentation. So, for the STAF facility the ROI = (4 live fires & 20 missiles/month 
for 12 months) / facility and instrumentation costs = $1.58 M / $5.8 M = .2724. This 
represents the annual ROI for the STAF facility. 

6. Step 6. Develop a Migration Strategy 

Of the four VPG projects examined in this thesis, only the use of the STAF 
facility was fully integrated into its associated acquisition program. The STAF facility is 
an integral part of the lot acceptance program for the Longbow missile. Unfortunately, 
formalized management goals and a systematic approach to capturing the expected 
benefits of the program were not developed. Based on this lack of formalized goals, the 
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monitoring and feedback process is limited to the future possibilities for use of this 
technology for program savings in general. 

7. Step 7. Monitoring the Process and Assessing Results through 
Formalized Feedback 

Other Army programs have been approached about using the STAF facility on a 
time-sharing basis and using HWIL simulation to reduce testing costs. The key factor 
here is the need to highlight the necessity of early planning to accomplish this task. 
LBHF has already demonstrated that it is much easier to design features into the missile 
that lend themselves to aiding the ease of HWIL testing, than attempting to correct these 
after the missile design has been locked in. In the case of LBHF, the move to HWIL 
simulation meant determining an after the fact means to attach the simulation hook up 
cable through a sealed access panel in the missile body. In addition numerous signal and 
interface problems and modifications were needed. These requirements would have been 
easier to resolve if they had been included in the original system design to accommodate 
a HWIL approach. Future systems should investigate this area during the design phase, 
so those HWIL critical features can be incorporated into the design. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The management of acquisition programs is one of the most difficult jobs in the 
DoD today. When making complex technological decisions where there are a limited 
number of variables but a near limitless combination, using a business case can help the 
decision-maker see through the fog. M&S allow the PM to experiment with a larger 
number of possibilities without undue risk and the T&E practitioner more opportunities 
to provide better testing at a lower cost and schedule. Once the essential data is 
determined, collected, and then shaped within a model, the simulations provide a 
tremendous advantage over traditional methods of trial and error. 

The use of M&S is no longer reserved for programs on the cutting edge of 
technological development, but rather it is standard practice. Since M&S is a part of 
everyday practice, both industry and the Government should rely on sound business 
practices to ensure the successful application of M&S. Based on the fact that M&S is an 
established business tool, M&S investment justification should be based on a reasoned 
cost benefit analysis. 

Establishing these cost benefit relationships can however be just as difficult as the 
management of a program itself. The use of the business case process contained in this 
thesis to justify M&S development provides a flexible yet structured methodology for 
decision-makers to weigh alternatives. This business case analysis permits the decision¬ 
maker to justify investment decisions based on cash flow analyses, as a function of 

externally imposed constraints, and risk reduction. Additionally, the business case allows 
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the capture of not only those costs and benefits that are internally quantifiable but also 
those that are non-quantifiable with respect to the potential benefits that may exist 
external to the program. 

When the business case was applied to four developmental testing historical 
cases, some poignant discoveries were made. When a structured, disciplined process as 
outlined in this thesis was applied to the Enhanced Position Location Report System 
(EPLRS), Advanced Chemical Agent Detector Alarm (ACADA), ABRAMS M1A2 and 
the Simulation Test Acceptance Facility (STAF) facility study data, many inadequacies 
were observed. Of the four cases analyzed, the STAF facility was the closest to using a 
business-oriented analysis. 

The initial baseline step of the business case process proved to be limited in the 
definition of metrics. These performance measures for the four cases were primarily 
limited to test time and test cost (equipment and labor). There was no clear statement of 
assumptions and constraints. The baseline did however address areas that were 
impossible or difficult to test physically. This initial step in the business case vividly 
displayed that there truly was no structured planning for the development of these test 
and simulation technologies. 

The vision and direction step of the business case process is marginally well 
developed by the Virtual Proving Ground Master Plan. This plan however does not 
establish any performance criteria or measures and there are no definite goals that are tied 
directly to any funding. Additionally, the plan is very high level in nature without any 
specific implementation mechanism or projects. The plan does set the vision for using 
M&S to transform developmental testing. 
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The steps of the business case that quantify the costs and benefits of alternatives 
and perform the analysis showed that a cost benefit analysis was performed in order to 
make the original decision to develop the STAF facility. Unfortunately, the other three 
cases did not perform a documented cost benefit analysis. In the case of the EPLRS, 
AC AD A and the ABRAMS M1A2, it was assumed that M&S must be used due to policy 
mandates and technology advances. The use of M&S in the EPLRS testing provided 
documented personnel multiplication factors that resulted in a test time reduction. The 
ACADA case used M&S where physical testing was not possible. Additionally, using 
M&S to test the ACADA proved to increase the number of units that could be tested 
simultaneously with fewer personnel. The ABRAMS M1A2 testing using simulation 
provided a cost avoidance for the quantity of personnel required, a reduction in test time 
and savings from not firing as much live ammunition. The VPG cost avoidance 
estimated for these programs are EPLRS $2.4M, ACADA $2.6M, ABRAMS M1A2 
$4.8M and realized for the STAF $8M. Other tangible benefits include: EPLRS 
shortened testing by 75 days, ACADA avoided 471 test days, ABRAMS M1A2 realized 
a schedule reduction of 258 days, and the STAF facility realized a facility payback period 
of one year. The STAF facility enables all of the live missiles tested to be put back in the 
inventory due to its non-destructive methods. Additionally, the STAF facility is able to 
test missiles under more rigorous and controlled conditions with fewer resources in less 
time than typical live firing programs. 

The conduct sensitivity analysis step of the business case showed that the STAF 
facility was the only case that used any type of sensitivity analysis to determine courses 
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of action. This step is critical to comparing the options and making a good business 
decision. 

The migration strategy development step of the business case is where the 
implementation plan is developed to enable the technology or project to be integrated into 
the acquisition program. This implementation plan needs to incorporate a systematic 
approach to ensure the developer implements the identified drivers and captures the 
expected benefits to be assessed at the post implementation review. Of the four projects 
analyzed, the STAF facility was the only project that was fully integrated into its 
associated acquisition program from inception. The methodology with virtual targets 
used in the Abrams M1A2 example is now fully integrated into all fire control testing for 
combat vehicles which is a test process integration. This type of test process integration 
is a goal of the Virtual Proving Ground. The use of the STAF facility to perform lot 
acceptance testing of future missile lots continues to be integral to success of the 
Longbow missile program. This integration was planned during the early design stages 
of the missile program. This early integration enabled and ensured that the STAF facility 
would successfully support the Longbow program. Additionally, the benefits of the 
implementation and the integration of the facility with the acquisition program are 
continually being tracked for future development of a STAF facility that supports other 
missile sensor technologies. The early planning and migration of the STAF facility has 
ensured its success and the future of the HWIL technology implementations for new 
programs. 

A whole generation of future tactical and operational level missile systems could 
all benefit from this type of lot acceptance testing. The major concern here is that these 
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systems need to make decisions about simulation testing now while still in their 
developmental stages. It has already been discovered that as these systems increase in 
complexity, the need to design in access and compatibility with simulation testing up 
front is crucial. Delaying this activity will only create additional design, interface, and 
cost problems. The future of testing will use simulation assets that will provide benefit to 
acquisition programs. This benefit can only be realized through proper early planning in 
the design of both development and testing systems. For example, this could be achieved 
by providing high fidelity system simulations early on for testing. 

The monitoring and feedback step of the business case, Step 7, is intended to 
monitor and assess the project’s progress towards the overall vision. The STAF facility 
is the only project that has monitored their process, determined ways to improve the 
process and then continued towards developing future facilities based on their concepts 
for using HWIL technologies to improve the testing capabilities of an acquisition 
program. By making an investment in current and future testing technology and applying 
it today, avoidance in testing costs and resources can be realized using virtual testing 
techniques. 

B. RECOMMENDATIONS 

Having a clear understanding of the state of development, the following 
recommendations will serve to assist decision-makers with the development of M&S 
investment strategies based on a sound business case development processes. 

PMs must be encouraged to add discipline and structure to their M&S justification 

process. Service leadership must challenge PMs to use a business case development 
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methodology to support M&S investment decisions. Decision-makers and their staff 
need adequate training in order to properly implement business case-based M&S 
investment strategy justification. Additionally, training in the development of strategic 
management techniques and practices would ensure that the vision step of the business 
case strategy is developed from an issue perspective with realistic objectives. 

When implementing the business case process, I recommend several points be 
kept in mind to ensure success. Detailed baselines must be developed with future data 
collection efforts in mind. The assumptions should include estimated future workload, 
useful life of the investment, and the period of time over which alternatives will be 
compared. The data collection activities will be used to justify and compare programs 
properly. Additionally, metrics to be used throughout the program should be realistic and 
tied to the program visions and goals. Total Ownership Cost (TOC) must be addressed 
more rigorously for all programs from concept through disposal. Too often TOC is 
ignored or inadequately addressed during the analysis. Rigorous reviews of the M&S 
effort must be performed throughout the program with continuous feedback to all 
elements of the program being developed and the process outputs recorded for the benefit 
of future development programs. The output of the Post Implementation R.eview (PIR) 
should feed into annual performance plans which will support the vision and direction set 
forth in the master or strategic plan developed. The annual performance plans should 
show how all of the functional requirements and the feasibility of the cost and benefits of 
the alternative will be tied to program schedules and budgets. 

Future VPG program and project efforts need to be based on a detailed plan which 
is embraced by all levels of management, which is supported by an annual performance 
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plan, and outlines the metrics by which the individual projects will be assessed. The 
performance plan should tie all project requirements to a funding line in the budget. All 
future major VPG projects which support the development of the DTC stand-alone 
capability and those that are tied to acquisition programs should be required to follow the 
business case analysis contained herein to ensure that the development effort is cost 
effective and meets the technical and business goals of DTC. 
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APPENDIX A. M&S AND CONTRIBUTING WORLD 
TECHNOLOGY EVENTS 


Year 

M&S and Contributing World Technology Events (In Chronological 
Order) 

1867 

Rigid Kriegspiel 

1917 

Ruggles Orientator 

1918 

Sand Tables 

1920 thru 

Tactical Research (Lindell Hart) 

luSsHHHi 

First Link Trainer 

1930 thru 

1940 

Wargames at Naval War College 


Navy Buys Instrumented Trainer ($1500) 


Army Buys 6 link trainers @ $3500 each 


German Soldiers stand guard with broom sticks simulating rifles 


Turing describes “universal machine” 

HBI 

Louisiana Maneuvers 


Bleetchley Park 


University of Illinois Aviation Lab 


Colossus Computing Machine (cryptography) 


Eniac (University of Pennsylvania) 


LINK Founded 


Whirlwind Digital computer 


Operations Research Formalized - Eisenhower Letter 


Transistor Patented 

1950 thru 

1960 

Contact Analog Display (George Hoover, USN) 


Link begins development of first simulator for fighter aircraft 


AAI and Texas Instruments Founded 


Flat See through display for aircraft 


ILLIAC (University of Illinois) 


First commercially available computer (Univac 1) 


Live/Virtual Simulations of Air Defense 


FAA Air Traffic Control Center for Simulations (FAA/RAND/SDC) 


First COEAs 


TI Produces Silicon Transistor 


Navy disposes of WWII aircraft - Generally Trainers 


DEC Founded 


FORTRAN 


SAGE System 


Kwajalein Test Site Selection 

. 

Minsky & McCarthy form AI Department at MIT 


101 

























First Mini Computer PDP-1 
LISP 1.5 

1960 thru First Triangulated Irregular Networked Terrain 
1970 

IBM 7070 

_ ALGOL 60 _ 

Line of Sight Algorithms 
NATO Range Study 
Lockheed Combat Model 
Carmonette Development 
Early Digital Terrain 

Teletype Model 33 with Punched Paper tape 
Simscript (RAND) 

~ FORTRAN IV 

Model Board Visual Systems 

Image Generator for NASA Docking Simulation (GE) 

Navy Transfers Kwajalein to Army 
First Mouse 
~ BASIC 

Lunar Lander Simulation (GE) 

TACOS Model (Air Defense) 

First Commercial Nuclear Power Plant Simulation (Indian Point) 

First Head Mounted Display (Ivan Sutherland) 

Evans & Sutherland Founded 
~ SIMULA 

Computer Image Generator (CIG) for Navy (GE) 

First Laser Application for Direct Fire Weapons 
Dyntacs Development 

Land Combat Systems Study LCS-90 using Carmonette 
AF Human Resources Lab Aircrew Research Established 
SAIC Founded 

ARPA Begins Development of ARPANET _ 

1970 thru First Advanced Simulator for Undergraduate Pilot 

1980 _ 

First Non-Realtime Demo of CIG Potential for Airline Training 
(Hancock Airport) 

Navy Establishes Top Gun 
~ SEL 86 

Air Combat Maneuvering Range and Instrumentation (ACMR/ACMI) 

AMC 71 Ground Mobility Model _ 

TACWARS Development Begins 
Royal Navy School of Training Established 
Intel Founded 

ATOWL Mission Planning Software 
TETAM Intervisibility Experiment 
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LCOM (TAC/LOG Manning) _ 

First Micro Processor Chip (Intel 4004 60,000 operations per sec - 

$ 200 ) _ 

First Email Demo _ 

Concepts Evaluation Model 
First USAF Helicopter Simulation 

AMSWAG _ 

LCMS (Log Capability Measurement System) _ 

ATARI Founded and ships “PONG” __ 

C Prolog _ 

Air Force Establishes Red Flag _ 

Mil-Std-1558 (Defacto 6-Post Motion System Standard) 

CATTS (Combined Arms Tactical Training System)_ 



DIVWAG 


First Electronic Calculator (TI) 


Packet Networking Developed 



Intel 8080 


Computer Aided Operations Research Facility - Kings Point 


Two Networked Full Motion Wide FOV Flight Simulations 


ARI Report on Tactical Engagement Simulation (TES) 

~ DIVLES 

Cope Thunder Exercise (TI) _ 

Gates & Allen Form Microsoft _ 

First Ethernet At Xerox _ 

_ gj mu j ator fidelity Studies (Beginning with motion/Without motion) 
AFHRL 

AIMVAL/ACEVAL Test 
MINI-3 (LLNL) 

_ VECTOR _ 

TACSM (Artillery) _ 

Jobs & Wozniak Form Apple and Produce Apple 1 
~ TRS-80 • 

NATO Reference Mobility Model _ 

Initial Concept for Large Scale Distributed Simulator Networking 
CIG Dome (VTRS) 

PTOS (Patriot Air Defense Simulation) _ 

ART BASS Studies _ 

AFOTEC uses simulation to identify T&E issues and extrapolate results 
~ JANUS (LLNL) ~ 

TAC BRAWLER Development Begins _ 

TRS-80 Calculates Battle Damage at Naval War College _ 

“War in Europe” Game _ 

Reforger ’79 driven by manual game “battle” _ 

Motorola 68000 
DEC 11/780 
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Intel 8086 

Table Top Gunnery Trainer 

1980 thru AMRAAM Test (four linked simulations) 

1990 __ 

First Class Cockpit Simulator (CAE Link8767 & MD80) 

_ Army National Training Center (NTC) 

First PC based Constructive Simulation (Wang 2000) 

VIGS (Video Disc Based Gunnery) 

Tank Driver Simulator ~ 

Navy Mobile Sea Ranges 
IBM PC 

Hayes Smart Modem 200 

_ Common Lisp " 

Four Low Cost Tank Gunnery Trainers Networked 
VCASS Helmet Mounted Display 

_ ARPA Video Arcade Tank Gunnery Trainer ($.25 per play) 

Mission Training Plans 
JANUS deliver ed to Army 

Silicon Graphics Formed ~ 

Geometry Engine (SGI) 

__ 

____ CRAY XMP _ 

Postscript 

TCMPAdopted ~ 

ADA 

_ Lotus 123 __ 

Delta Graphics Founded 

Warrior Preparation Center Established 

CAMMS (Munition in a minicomputer) 

ACA Buy (Urban Game) 

Casteforem Development 

ARPA SIMNET - Concept Validation Study 

MODSIM (Modular design for simulations) 

Compuscene III with Texture (GE) 

Air Force Established Blue Flag 
~ JANUS 4.0 

“Fire in the East” Board Game 

Apple Eye (Intervisibility on a microcomputer) 

VPL Founded (Jason Lanier) 

MAC 128 

ARPANET Split -MILNET Added (Later DDN) 



1,000 Most Computers on the net 


Apple LaserWriter 


Motorola 68020 
FATS Founded 


104 






























_ EOSAEL _ 

_ Bl-B Engineering Research Simulation _ 

_ IPFN Testbed (ALBQ) netted to PTOS simulation (El Paso) via T1 

JRTC Development and Demo 

_ Navy Fallon Tactical Aircrew Combat Trainer System (TACTS) 

METT-CAP Mission Planner 
JESS 

VIC (Vector in Commander) 

CBS Development Begins ~ 

_ Battle Command Training Program (BCTP) established using JESS 

Delta Graphics acquired by BBN ~ 

Motorola 68030 

NSFNET Created 

PERL 

Intel 80386 

JANUS TRAC 

_ SEES 1.0 (LLNL) 

_ Air Force Red Flag measurement and debrief system 

_CMTC Opens 

_ IRIS 4D70G with MIPS R2000 (First Unix RISC workstation (SGI) 

_ TAC Evaluation of Limited FOV Visual (FI 5) 

_ Semi-Automated Forces (SAF) Development Begins 

First Networked Simulations - Ft. Knox 

_ First Z Buffered Low Cost Multi-Channel CIG (Delta Graphics) 

Multigen 1.0 
' SIMNET-D 

_ SIMNET Graphics Engine (Delta Graphics) 

_ Compuscene PT 2000 (GE) 

OPTEC FOG-M Comparative Test ~ 

MS Excel 

C++ 2.0 (multiple inheritance) . 

10,000 Most Computers on the Net 

_ “Morris Worm” Attacks 7,000 of 60,000 Most computers on the Net 

NSF Backbone to T1 (1.5 MB/sec) 

_ViewPoint Data Labs Founded 

Strategic Defense Simulation and Performance Assessment 

_ Brave Shield “88 - First JWC Distributed CBS Exercise 

EADSIM Development Begins 

First Formal User Test of SIMNET (Ft. Knox to Hood) 60 Netted 
Simulations 

_ Distributed Interactive Simulation (PIS) Workshops Begin 

_ First Long Haul Training Exercise (Ft. Knox to Rucker) _ 

SIMNET Testbed, 250 Simulations, 11 Sites 
1990 thru BFIT (SIMNET to Naval Ship) 

1995 
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VP 1000 First PC Based CIG (TSI) _ 

Extended Air Defense Test Bed (EADTB) Begins _ 

Multiship R&D Testbed 

Marksmanship Trainers at 100 locations (USMC) 

British Army Approves Simulation Strategy 

Crossbow SA-12 Digital Simulation Prototype _ 

Simulation Policy Study (OSD/FMP/TRF) 

ARPANET Transitioned (Ceases) 

Visual Basic ~ 

Sun Sparc 2 
DMSO Established 
Maritime ITEM Development Begins 
First Aggregate Level Simulation Protocol (ALSP) Demo 
' JCM (LLNL/JWC) 

' JMASS Established _ 

Unit Performance Assessment System (UPAS) 

RADSIM 1.0 (Low Cost Radar Simulation) 

CD ROM Map Libraries 
SIMNET SAF 6.1 
NPSNET 

Mobile SIMNET (ANG) 

First DIS 1.0 Demon (TSI) 

Close Combat Tactical Trainer (CCTT) ASARC 

Battlefield Distributed Simulation (BDS-D) and Advanced Distributed 

Simulation Tactical (ADST-I) 

ALSP ULCHI Focus Lens Europe-US-Korea _ 

First Executive Council Modeling and Simulation (EXCIMS) meeting 
MIPS R4000 64 Bit RISC 

NSFNET BackBone to T3 (45 MB/sec) _ 

GOPHER (University of Minnesota) 

BICM (Battlefield Intel Collection Model) 

Defense Science Board (DSB) Report on Advanced Distributed 
Simulation (ADS) 

World Wide Web (WWW) application (CERN) 

STRICOM Established 
CSSTSS Development Begins 
Reforger ’92 First Simulation Driven FTX 
ALSP Program Funded 
JCM/SOFNET (LLNL/JWFC AF/DMSO) 

SEES (Security Exercise Evaluation System 
Marksmanship Trainer On Board (USMC) 

First C-17 ATS 

Compuscene VI Target Generation (GE) 

MOD SAF 
VRLink Ships 


106 



































SIMNET to CSRDF 
‘ CCTT AWARD 

' IITSEC Exhibit Floor “On the Net” _ 

Warbreaker 

Joint Precision Strike Demonstration (JPSD) 

SGI Reality Engine _ 

First Onyx (SGI) 

Topscene on Aircraft Carrier 

Automated Training analysis and Feedback System (ATAFS) 
Simulation Training Integrated Performance Evaluation System 
(STRIPES) 

' JMASS 1.0 _ 

Core Battle Simulation (CBS) STAARS 

Joint WarFighting Center(JWFC) Standup 

BCBST (Battle Command and Brigade Staff Trainer) (BBS and 

JANUS) _ 

Simulation Demo to SASC 

Intel 60 MHZ Pentium _ 

1,000,000 Host Computers on the NET 
' MOSAIC “Surf the NET” 

WWW Worms, Crawlers, Snakes, Spiders collect information around 

the WEB _ 

Motorola Power PC 601 _ 

Martin Marietta Buys GE Aerospace 
Loral Buys BBN Sim + Quintron 
AF/XOM Established 
GPS Based Ranges 

AFRES (Multi-Task Trainer Low Cost F-16 unit level) 

USAF Visual System Evaluation 
Synthetic Theater of War (STOW) 

ADS Architecture Study 
ZERO Regard 
CFOR CSSIL 

MDT2 Multi Service Distributed Trainer Testbed 
PIS Protocol 1.0 Approved (IEEE 1278) 

INCOMMS ’94 First Demo of Dismounted unit (virtual) netted to 

brigade (constructive) - Ft.Benning 

TACCSF 

First DIS Voice communications (TSI) 

A2ATD (Air Defense and Armor) 

TADSIM (Theater wide DIS R&D) 

Infoscope (TSI) 

JCOS 

Leathemet (USMC) _ 

Zealous Pursuit 
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IPORT 1.0 (Unicycle) 


High Level Architecture (HLA)/Architecture Management Group 
(AMG) 


STOW-Europe 


MAIS-DIS Compliancy Demo 


MORS ORSA Handbook 


MORS SimVal Symposium 


Joint ADS JT&E (JADS) - T&E Evaluation 


JCM/SIMNET (STRICOM) 


Atlantic Resolve 


JTLS 1.85 (First Joint Multi-sided Simulation) 


BIDS 


Loral Buys IBM Federal Systems 


MAC Power PC 


DEC Alpha 300 Mhz 


GPS Full IOC 


Netscape IPO 


7,000,000 Host computers & 61,000 Networks on the NET 


VRML 


Windows ‘95 


Lockheed and Martin Marietta Merge 


DMSTTIAC Established 


DoD M&S Master Plan 


BIDS (LUT/DPG/UT) 


NTF Wargame 


Roving Sands 


JWID ‘95 


Kernel Blitz 


PSD ‘95 


Prairie Warrior 


MOBA Digital Data (Benning & Quantico) 


MILES 2000 


Small Arms Naval Trainer 


HLA Protofederations 


PORT 2.0 (Treadmil) 


BBS-MODSAF 


ADST-II 


Table 1. M&S and Contributing World Technology Events after (Thorpe, 1995) 
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APPENDIX B. SIMULATION BASED ACQUISITION (SBA) 

“CHEAT SHEET” 

Tasking : Lt Gen Martin suggested that SARDA develop a “cheat sheet” that he 
and other decision-makers can use to ask program managers the “right questions.” The 
purpose of this is two-fold: first, to educate the decision makers on what the philosophy 
of SBA actually means when it comes to implementing it on programs, and second, to 
cause the program managers to “answer the question before its asked” (as Lt Gen Martin 
said) and in effect, start implementing SBA on their programs in the process of preparing 
to answer the questions. Following are (1) general modeling and simulation questions, 
and (2) SBA - specific modeling and simulation questions. 

A. GENERAL M&S QUESTIONS 


1. M&S Planning 

• IS THERE A PLAN FOR M&S USE ACROSS THE SYSTEM LIFECYCLE? 

• IS THE M&S STRATEGIC PLAN INCLUDED IN THE ACQUISITION 
STRATEGY DOCUMENT? IS IT UPDATED ANNUALLY? 

• IS THE PLAN FUNDED? 

• DOES THE PLAN TAKE INTO CONSIDERATION: 

• IDENTIFICATION OF ALL MODEL AND SIMULATIONS 

• THE SOURCE OF NEEDED DATA FOR THE MODELS AND SIMULATIONS 

• REUSE ACROSS ACQUISITION PHASES? 

• VV&A? 

• USE OF MODELS IN SOURCE SELECTION AND THEIR RELEASE TO 
INDUSTRY? 

• IS THE USE OF M&S ACCOUNTED FOR IN THE TEMP? 


2. Requirements 


• WHAT MODELS/SIMS AND DATA WERE USED, IF ANY, AS PART OF THE 
AOA TO GENERATE REQUIREMENTS? WERE THEY VERIFIED, VALIDATED 
AND ACCREDITED? 


3. Cost 


• DOES THE FUNDING PROFILE SUPPORT THE ROBUST USE OF M&S 
THROUGHOUT THE PROGRAMS LIFECYCLE? (E.G., FRONT-LOADED?) 

• DOES THE USE OF M&S MAKE THE PROGRAM MORE AFFORDABLE? IF 
SO, IN WHAT WAY? 

• COST AVOIDANCE 

• COST SAVINGS 
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• SCHEDULE SAVINGS 

• OTHER 

4. Contractor Incentives 

• ARE THE PROPER INCENTIVES IN PLACE FOR THE CONTRACTOR TO USE 
A DIGITAL ENVIRONMENT TO CONTAIN COSTS? 

B. SBA-SPECIFIC QUESTIONS 

SBA is a new acquisition strategy that exploits recent advances in M&S 
technologies. It involves not only changes in the technologies and environment, it also 
involves changes in the culture and processes. The following questions seek evidence of 
those changes. 

SBA enables a more iterative acquisition process versus the traditional serial 
process of moving through the phases. We conceive, design, manufacture, test, train, 
operate & sustain (in an iterative nature) concurrently in the virtual world before locking 
in on a solution. We delay locking in our requirements (ORD) until we examine the trade 
space in a virtual system of systems environment. As the system is developed and 
produced, the representation of the new system continues to be matured in parallel with 
that development. In terms of implementation, this process should be reflected, in part, in 
the program acquisition strategy in a section entitled “M&S approach”: 

1. Shared Digital Environment - SBA Concepts That Apply 

• Collaborative Environments - Simply put, this is 2 or more programs collaborating 
together on a given problem and sharing and reusing models/sims. Example: missile 
PM collaborating with aircraft and ship PMs for a strike warfare collaborative 
environment; or, several aircraft PMs collaborating across the aircraft product line 

• QUESTION : IS THE PROGRAM PARTICIPATING IN ANY COLLABORATIVE 
ENVIRONMENTS? 

• Distributed Product Description or DPD (e.g., virtual representation of the system 
including all the associated information that makes it “smart” such as the function of the 
system, special manufacturing requirement, cost of system, etc. a DPD is a step beyond 
a virtual prototype) 

• QUESTION : DOES THE ACQUISITION STRATEGY CALL FOR 
DEVELOPMENT OF A DPD? IF SO, WILL THE GOVERNMENT OWN THE 
RIGHTS? IS IT PLANNED TO EVOLVE THE DPD OVER THE LIFE OF THE 
PROGRAM? 

• DoD/Industry Resource Repository or DIRR - A shared repository for dpds as well as 
other models and simulations to enable government and industry sharing and reuse. It 
is an extension of the existing DoD modeling and simulation resource repository 
(MSRR). Access controls required - example: Boeing may have a Boeing-proprietary 
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DPD of the JSF that only Boeing and the government can access (but not Lockheed). 
JSF may also use a “purple DPD” that anyone can access. 


• QUESTION : DOES THE ACQUISITION STRATEGY CALL FOR PLACEMENT 
OF THE DPD IN THE MSRR (FUTURE -DIRR)? HAVE APPROPRIATE ACCESS 
CONTROLS/PROPRIETARY DATA RIGHTS ISSUES BEEN RESOLVED? 

• Ultimately, when SBA is implemented as a strategy, programs, working in 
conjunction with each other, will use common standards and adopt a common 
architecture which will facilitate interoperability and reuse supporting DoD goals of 
reduction in cycle time and cost. 


• QUESTION : HOW IS THE PROGRAM WORKING TO LEAVERAGE EXISTING 
STANDARDS (E.G., HLA)? 

• QUESTION : HOW ARE YOU USING STANDARDS TO ENSURE 

INTEROPERABILITY WITH OTHERS OUTSIDE YOUR PROGRAM? 

• QUESTION : WHAT M&S ARE YOU LEAVERAGING FROM OTHER 
PROGRAMS? 

• QUESTION : WHAT PARTNERSHIPS HAVE YOU INTIATED/INTEND TO 
SUSTAIN TO BOTH CONTRIBUTE TO THE COMMUNITY AND REDUCE 
DEPARTMENT ACQUISITION COSTS? 

2. Simulation Test and Evaluation Process (STEP) 


• STEP is an integral part of the SBA strategy. The STEP Guidelines, signed by Dr. 
Sanders and Mr. Coyle, require an evaluation strategy integrating test and evaluation 
(T&E) with M&S. 

• QUESTION : DOES YOUR PROGRAM HAVE AN INTEGRATED T&E AND 
M&S STRATEGY? 

• QUESTION : TO WHAT EXTENT IS YOUR EVALUATION STRATEGY 
IMPLEMENTING THE STEP GUIDELINES? 

• QUESTION : STEP REQUIRES EARLY INPUT FROM THE TEST COMMUNITY 
AS PART OF THE ACQUISITION LIFE CYCLE. AS PART OF YOUR 
EVALUATION STRATEGY, AT WHAT POINT DO THE TESTERS BECOME 
INVOLVED IN YOUR PROGRAM? 

• QUESTION : WILL OT USE M&S FOR IOT&E? 
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3. Evolved Process 


• Integrated Product And Process Development (IPPD) and the use of Integrated 
Product Teams (IPTs) with empowered government and industry members to 
accomplish defined tasks are critical to take advantage of the technological advances 
which make SBA possible. 

• QUESTION : HAS THE PROGRAM FORMED GOVERNMENT/INDUSTRY 
IPTS, INCLUDING ONE FOR M&S, WITH EMPOWERED GOVERNMENT AND 
INDUSTRY MEMBERS TO ACCOMPLISH THE DEFINED TASKS THAT ARE 
CRITICAL TO TAKE ADVANTAGE OF THE TECHNOLOGICAL ADVANCES 
WHICH MAKE SBA POSSIBLE? 

4. Evolved Culture 

• Incentives are required for development in the initial stages of a program to develop 
the integrated data environment and other infrastructure and resources for SBA. 


• QUESTION : HAS THE PROGRAM MANAGER PROVIDED INCENTIVES FOR 
INDUSTRY TO EITHER ASSIST IN OR DEVELOP THE NECESSARY 
PRODUCTS AND SERVICES FOR SBA TO BE IMPLEMENTED AS A 
STRATEGY? 


• The process must build a partnership between government and industry. The first 
opportunity of this is during the request for proposals and proposal evaluation 


• QUESTION : DOES THE ACQUISITION STRATEGY CALL FOR THE SHARING 
OF MODELS & SIMULATIONS EARLY ON (ALA IPPD), TO INCLUDE DURING 
SOURCE SELECTION, AND DURING THE LIFE OF THE PROGRAM? 


(SARDA, 2000) 
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APPENDIX C. COST FACTOR DEFINITIONS 


A. COMMUNICATIONS - NONRECURRING 

Total nonrecurring expenditure of communications equipment and services to 
make the fully configured and installed system operable at its inception. Include in this 
category data communications equipment, such as modems and data encryption devices, 
as well as other communications costs, such as local area networks. 

B. COMMUNICATIONS - RECURRING 

Yearly payments for communications costs. Include in this category data 
communications equipment, such as modems and data encryption devices, as well as 
other communications costs, such as local area networks. 

C. CONTRACTING 

Total expenditure of contracts for construction, design, development, consulting, 
and installation of system. 

D. CONVERSION COSTS 

For replacement or upgrade systems, the incremental costs incurred only from the 
costs to convert hardware (such as PCs, mainframes, disk drives, servers, printers) or 
software (such as database, batch programs, and expert systems) from one system to 
another. These costs are only incurred if the system under consideration is replacing 
another specific system. Examples are batch transfer programs, data re-entry, hardware 
modification. 

E. EQUIPMENT 

Yearly costs allocated for the purchase, lease or rental of other equipment to 
support business operations excluding equipment associated with the new system. 


113 






F. EQUIPMENT - OTHER 

Yearly costs allocated for the purchase, lease or rental of other equipment to 
support the use of the system. Include in this category photocopiers, file cabinets, fire 
safes, microfiche, optical storage facilities and other office products. 

G. FACILITY SPACE OCCUPANCY 

Yearly payments for allocated rents and building user costs for business 
operations other than the space allocated to the new system (excludes utilities). 

H. HARDWARE-NONRECURRING 

Total dollar expenditure, by year, of hardware to make the fully configured and 
installed system operable at its design inception. Include in this category mainframes, 
desktop, laptop, PCs, disk drives, tape drives, display monitors, keyboards, printers and 
other peripheral equipment. This does not include possible conversion costs for 
upgrading from older systems. 

I. HARDWARE LEASE OR RENTAL - RECURRING 

Yearly payments for system hardware lease or rental. Include in this category 
mainframes, desktop, laptop, PCs, disk drives, tape drives, display monitors, keyboards, 
printers and other peripheral equipment. 

J. INCIDENTALS 

Other minor costs associated with day to day business operations. Exclude 

incidental costs associated with the new system. 

K. INTERAGENCY 

Yearly payments to (less credits received from) other agencies for shared facilities 
used and services other than systems operations. 
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L. MAINTENANCE 

Yearly payments for maintenance of business operations. Exclude from this 
category all repairs, maintenance, and emergency service support costs directly 
attributable to the system. 

M. PARALLEL OPERATIONS 

Expenditure needed for parallel operations or systems testing on a nonrecurring 

basis. 

N. PERSONNEL SALARIES AND FRINGE 

Yearly allocated costs for personnel who perform Benefits functions to be 
supported by the new system. This should include all organizational units and sites whose 
operations will change as a result of the system. 

O. RESIDUAL VALUE 

The salvage value of the entire system at the conclusion of its life cycle. 

P. SECURITY 

Yearly costs to provide system security and integrity. Include in this category 
security monitoring systems, alarm systems, camera and voice recording and storage 
systems, lock and pass key systems, and security personnel costs. 

Q. SOFTWARE-NONRECURRING 

Total dollar expenditure of software to make the fully configured and installed 
system operable at its inception. Include in this category all system software packages, 
off-the-shelf software, and custom software, site or network license and all original 
software programming and development costs. Important: do not include software 
conversion or upgrade costs here; instead, include these in conversion costs. 
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R. SOFTWARE LEASE OR RENTAL - RECURRING 

Yearly payments for system software lease or rental. Include in this category all 
system software packages, off-the-shelf software, and custom software, site or network 
license, and all original software maintenance costs. 

S. STUDIES 

Total expenditure for studies necessary to implement this system alternative. 

T. SUPPLIES 

Yearly payments for supplies to operations other than those associated with the 
new system. 

U. SYSTEM TESTING AND BACKUP 

Recurring costs to test reliability and integrity of system in operation, including 
costs for memory back up. This does not include costs for nonrecurring start-up and 
installation testing. 

V. TRAINING 

Recurring training costs associated with routine business operations, e.g., training 
aimed at improving supervisory skills. Exclude costs for training associated with the new 
system. 

W. TRAVEL 

Yearly travel expenses incurred in the normal course of business operations. 
Exclude any travel costs that are associated with the new system. Include train, bus, taxi, 
and airline tickets, gas, mileage and toll charges and auto rental expenses. 

X. USEFUL SYSTEM LIFE 

Forecast of the planned useful system life from the first month of system 
implementation. 
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Y. UTILITIES 

Yearly payments for costs of utilities allocated to business operations other than 
the new system. 
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APPENDIX D. DOCUMENTATION CHECKLIST 


DOCUMENTATION CHECKLIST 

□ Alternatives Analysis: 

□ 

Benefits Cost Analysis (BCA): 

• Answer the Three Pesky Questions, 

• 

Assumptions and constraints 


• 

Alternatives and their schedule, costs, and benefits 

1) Does the investment in a major capital asset support 

• 

Risk and sensitivity analysis 

core/priority mission functions that need to be performed by 

• 

Performance goals and measures for monitoring the 

the Federal Government? 


project 

2) Does the investment need to be undertaken by the 
requesting agency because no alternative private sector or 

□ 

Feasibility Analysis and Market Research: 

government source can better supportthe function? 

• 

Feasibility: 

• Availability 

3) Docs the investment support work processes that have been 


• Affordable 

simplified or otherwise redesigned to reduce cost, improve 
effectiveness, and make maximum use of commercial, off-the- 


• Cost & Benefits 

self technology? 

• 

Market Research: 

• Availability of commercial items to meet the need. 



and whether they require modification 

□ Project Plan: 


• Distribution and support capabilities of suppliers 

• Major milestones with schedule 

• Deliverables 

o 

Functional Requirements Analysis: 

• Critical path decision points 

• Resources 

• 

Performance criteria, goal, or ultimate output 

• Integrated Project Team 

• 

Definition of the common uses of die IT investment 


• 

Ranking of each requirement in order of importance 


• 

Decomposition of functional requirements into self- 

□ Acquisition Plan - Part One : 


contained features 

• Statement of need 

• 

Requirements should be described in terms ofi 

• Significant conditions affecting acquisition 


• Business outcome 

• Capability or performance 


• Purpose 

• Tradeoffs 


• S/SO program components involved 

• Risks 


• Operating constraints 

• Modular contracting 


• Mission 

• Capability 

□ Baseline Assessment and Statement of Need: 


• Schedule and cost objectives 

• Baseline assessment criteria 

□ 

QA Plan: 

• Functionality 

• Full life cycle cost 

• 

Life cycle process 

• Expected funding levels cost 

• Security Plan 

♦ 

Periodic independent review 

• Capacity to manage asset 

• Existing resource baseline 

□ 

Risk Management Plan: 

• Performance gap 

• 

Risks including costs, schedule and technical 


• 

Potential risks 


Figure 6. Document Checklist from (GSA, 2000) 
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DOCUMENTATION CHECKLIST (CONT.) 

□ 

Security Plan: 

□ 

Disaster/Recovery and Contingency Plan: 

* 

Government information protected from misuse, loss and 

• 

Strategy for mission performance and recovery from 


unauthorized access 


loss of existing support 

• 

Security responsibilities assigned 

• 

Plans for continuous testing of the system 

• 

Systems security plan reviewed by security specialist 




Rules of the system 

□ 

Post Implementation Review: 

• 

Training issues 



• 

Incident response capabilities 

» 

System reviewed and documented on a regular basis 

• 

Interconnectivity security addressed 

• 

PIR conducted by an independent evaluation team 

• 

Security controls reviewed 

• 

Customeruser satisfaction reviewed 



• 

Internal business reviewed 

□ 

Acquisition Plan - Part Two: 

* 

Strategic impact and effectiveness reviewed 



• 

Innovation reviewed 

• 

Source of supply 



• 

Competition description 



• 

Source selection procurement 



• 

Contracting consideration 



• 

Budgeting and funding 



• 

Product description 

1 


• 

Priorities 



• 

Contractor vs. Government preference 



• 

Performance Management System 



• 

Test and evaluation 



• 

Logistics considerations 



» 

Indicate Government furnished property 



• 

Indicate Government furnished information 



□ 

Plans for Assets f IT Investments: 



• 

Operational analysis 



• 

Steady State Plan 



• 

Schedule Post Implementation Review 



• 

Asset Disposal Plan 




Figure 7. Document Checklist Continued from (GSA, 2000) 
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APPENDIX E. PROJECT RISKS CHECKLIST 


PROJECT RISKS CHECKLIST 


□ Strategic Risk 

Alignment with the agency's overall business strategy 
Clarity of expression of anticipated project outcomes. 

Presence of metrics to verify die successful completion of each project phase. 

□ Financial Risk 

Size of expenditure required. 

Existence of cost/benefit analysis. 

Existence of defined payback and time frame of payback. 

Reputation and financial status of vendor(s). 

□ Project Management Risk 

Experience of project management teams 
Existence of work plan for entire life cycle. 

Degree of development of measurable milestones. 

Length of time for project implementation. 

Existence of system for tracking unresolved issues. 

Definition of user and development skill requirements. 

□ Technology Risk 

Plan for validating that user needs are met. 

Existence of load test in accordance with industry standards 
Evaluation of technology options. 

Maintainability and ability to upgrade key technologies. 

Vendor's ability to implement technology. 

□ Security Risk 

Perform risk assessment 
Implement security controls. 

Security training and awareness. 

Contingency planning & disaster recovery. 

Comply with security policy. 

□ Change Management/Operational Risk 

Development of acceptance plan. 

Experience and ability of existing staff to support new system. 

Organization's familiarity with proposed hardware/software environment. 
Development of system operating procedures. 

Impact to organization of system failure. 

Magnitude of change introduced by system. 

Number of business units impacted. 


Figure 8. Risk Checklist from (GSA, 2000) 
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